[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 434668424 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003163] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.007899] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010142] LSM: Security Framework initializing [ 0.011054] Yama: becoming mindful. [ 0.013040] SELinux: Initializing. [ 0.014074] *** VALIDATE selinux *** [ 0.022759] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027796] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030133] *** VALIDATE tmpfs *** [ 0.031461] *** VALIDATE proc *** [ 0.032266] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.035127] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036150] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038035] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.042380] debug: unmapping init [mem 0xffffffffaca59000-0xffffffffaca60fff] [ 0.044270] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045778] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046023] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050012] ... max period: 00007fffffffffff [ 0.051013] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.054232] rcu: Hierarchical SRCU implementation. [ 0.056552] smp: Bringing up secondary CPUs ... [ 0.057648] x86: Booting SMP configuration: [ 0.058030] .... node #0, CPUs: #1 #2 #3 [ 0.062118] smp: Brought up 1 node, 4 CPUs [ 0.064018] smpboot: Max logical packages: 1 [ 0.065021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.271663] node 0 deferred pages initialised in 204ms [ 0.273189] devtmpfs: initialized [ 0.274339] x86/mm: Memory block size: 128MB [ 0.277960] gcov: version magic: 0x41383552 [ 0.281323] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.284102] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.287439] pinctrl core: initialized pinctrl subsystem [ 0.289257] [ 0.289736] ************************************************************* [ 0.292017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.294017] ** ** [ 0.296018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.299016] ** ** [ 0.301016] ** This means that this kernel is built to expose internal ** [ 0.303017] ** IOMMU data structures, which may compromise security on ** [ 0.305013] ** your system. ** [ 0.308016] ** ** [ 0.310014] ** If you see this message and you are not debugging the ** [ 0.312018] ** kernel, report this immediately to your vendor! ** [ 0.314015] ** ** [ 0.316019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.319017] ************************************************************* [ 0.321913] NET: Registered protocol family 16 [ 0.324552] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.326077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.329084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.333046] cpuidle: using governor menu [ 0.334740] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.335399] PCI: Using configuration type 1 for base access [ 0.336137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.343114] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.346040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.350038] cryptd: max_cpu_qlen set to 1000 [ 0.354073] ACPI: Added _OSI(Module Device) [ 0.356021] ACPI: Added _OSI(Processor Device) [ 0.358031] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.361021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.366911] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.373644] ACPI: Interpreter enabled [ 0.375081] ACPI: PM: (supports S0 S3 S4 S5) [ 0.376015] ACPI: Using IOAPIC for interrupt routing [ 0.378141] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.381506] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.391577] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.395050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.398031] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.401134] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.406524] acpiphp: Slot [2] registered [ 0.408166] acpiphp: Slot [5] registered [ 0.409172] acpiphp: Slot [6] registered [ 0.411155] acpiphp: Slot [3] registered [ 0.412107] acpiphp: Slot [4] registered [ 0.414131] acpiphp: Slot [7] registered [ 0.415334] acpiphp: Slot [8] registered [ 0.417127] acpiphp: Slot [9] registered [ 0.418105] acpiphp: Slot [10] registered [ 0.420202] acpiphp: Slot [11] registered [ 0.421121] acpiphp: Slot [12] registered [ 0.423131] acpiphp: Slot [13] registered [ 0.424145] acpiphp: Slot [14] registered [ 0.426143] acpiphp: Slot [15] registered [ 0.428123] acpiphp: Slot [16] registered [ 0.429104] acpiphp: Slot [17] registered [ 0.431111] acpiphp: Slot [18] registered [ 0.432124] acpiphp: Slot [19] registered [ 0.434121] acpiphp: Slot [20] registered [ 0.436123] acpiphp: Slot [21] registered [ 0.437131] acpiphp: Slot [22] registered [ 0.439126] acpiphp: Slot [23] registered [ 0.440114] acpiphp: Slot [24] registered [ 0.442117] acpiphp: Slot [25] registered [ 0.444122] acpiphp: Slot [26] registered [ 0.445114] acpiphp: Slot [27] registered [ 0.447140] acpiphp: Slot [28] registered [ 0.448113] acpiphp: Slot [29] registered [ 0.450114] acpiphp: Slot [30] registered [ 0.451119] acpiphp: Slot [31] registered [ 0.453080] PCI host bridge to bus 0000:00 [ 0.455024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.457043] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.459030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.462276] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.465035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.468040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.470198] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.472970] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.475079] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.481359] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.487770] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.490025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.492025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.494025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.497593] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.500865] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.504056] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.506833] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.512009] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.521022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.524961] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.529300] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.538022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.547022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.565018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.576214] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.585019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.591021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.602020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.612642] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.615490] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.618411] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.621428] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.623253] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.629049] iommu: Default domain type: Passthrough [ 0.630449] SCSI subsystem initialized [ 0.632105] ACPI: bus type USB registered [ 0.633125] usbcore: registered new interface driver usbfs [ 0.635117] usbcore: registered new interface driver hub [ 0.637092] usbcore: registered new device driver usb [ 0.638173] pps_core: LinuxPPS API ver. 1 registered [ 0.640017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.642063] PTP clock support registered [ 0.644049] EDAC MC: Ver: 3.0.0 [ 0.646128] PCI: Using ACPI for IRQ routing [ 0.647524] NetLabel: Initializing [ 0.649012] NetLabel: domain hash size = 128 [ 0.651025] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.653100] NetLabel: unlabeled traffic allowed by default [ 0.655123] vgaarb: loaded [ 0.656244] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.658025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.665000] clocksource: Switched to clocksource kvm-clock [ 0.756743] VFS: Disk quotas dquot_6.6.0 [ 0.757843] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.759733] *** VALIDATE ramfs *** [ 0.760615] *** VALIDATE hugetlbfs *** [ 0.761736] pnp: PnP ACPI init [ 0.763623] pnp: PnP ACPI: found 6 devices [ 0.779556] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.781774] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.783263] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.784650] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.786185] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.787729] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.789618] NET: Registered protocol family 2 [ 0.791432] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.794864] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.797398] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.801293] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.804113] TCP: Hash tables configured (established 65536 bind 65536) [ 0.806241] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.808382] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.810337] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.812407] NET: Registered protocol family 1 [ 0.814422] RPC: Registered named UNIX socket transport module. [ 0.815807] RPC: Registered udp transport module. [ 0.816911] RPC: Registered tcp transport module. [ 0.818178] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.819887] NET: Registered protocol family 44 [ 0.821224] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.822767] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.823970] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.825677] PCI: CLS 0 bytes, default 64 [ 0.827643] Unpacking initramfs... [ 2.143819] debug: unmapping init [mem 0xffff9a613cc64000-0xffff9a613ffcffff] [ 2.147764] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.149976] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.152709] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.644061] Initialise system trusted keyrings [ 2.645647] Key type blacklist registered [ 2.647788] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.656432] zbud: loaded [ 2.659602] *** VALIDATE nfs *** [ 2.660903] *** VALIDATE nfs4 *** [ 2.662566] pstore: using deflate compression [ 2.666576] Platform Keyring initialized [ 2.756960] NET: Registered protocol family 38 [ 2.758412] Key type asymmetric registered [ 2.760100] Asymmetric key parser 'x509' registered [ 2.762103] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.764248] io scheduler mq-deadline registered [ 2.765460] io scheduler kyber registered [ 2.766608] io scheduler bfq registered [ 2.767827] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.770044] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.771904] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.773799] ACPI: Power Button [PWRF] [ 2.777538] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.782341] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.789248] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.818067] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.848480] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.853971] Non-volatile memory driver v1.3 [ 2.855578] Linux agpgart interface v0.103 [ 2.888311] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.890474] vda: detected capacity change from 0 to 74702848 [ 2.902922] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.905463] vdb: detected capacity change from 0 to 1073741824 [ 2.910106] libphy: Fixed MDIO Bus: probed [ 2.930392] usbcore: registered new interface driver usbserial_generic [ 2.933147] usbserial: USB Serial support registered for generic [ 2.936106] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.941257] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.943450] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.945886] mousedev: PS/2 mouse device common for all mice [ 2.949345] rtc_cmos 00:05: RTC can wake from S4 [ 2.952317] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.952791] rtc_cmos 00:05: registered as rtc0 [ 2.958766] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.958957] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.962029] intel_pstate: CPU model not supported [ 2.968708] hid: raw HID events driver (C) Jiri Kosina [ 2.970408] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.971665] usbcore: registered new interface driver usbhid [ 2.976406] usbhid: USB HID core driver [ 2.977624] drop_monitor: Initializing network drop monitor service [ 2.980317] Initializing XFRM netlink socket [ 2.982672] NET: Registered protocol family 10 [ 2.985699] Segment Routing with IPv6 [ 2.987224] NET: Registered protocol family 17 [ 2.989430] mpls_gso: MPLS GSO support [ 2.995308] RAS: Correctable Errors collector initialized. [ 2.997732] AVX version of gcm_enc/dec engaged. [ 2.999645] AES CTR mode by8 optimization enabled [ 3.085195] sched_clock: Marking stable (3085165698, 0)->(4000719070, -915553372) [ 3.088351] registered taskstats version 1 [ 3.090222] Loading compiled-in X.509 certificates [ 3.091686] zswap: loaded using pool lzo/zbud [ 3.113968] Key type big_key registered [ 3.124880] Key type encrypted registered [ 3.126869] ima: No TPM chip found, activating TPM-bypass! [ 3.129370] ima: Allocated hash algorithm: sha1 [ 3.131078] ima: No architecture policies found [ 3.132834] evm: Initialising EVM extended attributes: [ 3.135015] evm: security.selinux [ 3.136436] evm: security.ima [ 3.137700] evm: security.capability [ 3.139042] evm: HMAC attrs: 0x1 [ 3.141573] rtc_cmos 00:05: setting system clock to 2026-08-15 08:31:48 UTC (1786782708) [ 3.148343] debug: unmapping init [mem 0xffffffffada03000-0xffffffffadbfffff] [ 3.151700] debug: unmapping init [mem 0xffffffffac782000-0xffffffffaca58fff] [ 3.159116] Write protecting the kernel read-only data: 28672k [ 3.162647] debug: unmapping init [mem 0xffffffffaae03000-0xffffffffaaffffff] [ 3.165743] debug: unmapping init [mem 0xffffffffab714000-0xffffffffab7fffff] [ 3.202437] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.211828] systemd[1]: Detected virtualization kvm. [ 3.213759] systemd[1]: Detected architecture x86-64. [ 3.216164] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.245237] systemd[1]: No hostname configured. [ 3.246633] systemd[1]: Set hostname to . [ 3.248403] random: systemd: uninitialized urandom read (16 bytes read) [ 3.250198] systemd[1]: Initializing machine ID from random generator. [ 3.413896] random: systemd: uninitialized urandom read (16 bytes read) [ 3.416835] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.421051] random: systemd: uninitialized urandom read (16 bytes read) [ 3.424067] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.436433] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ 3.599055] random: fast init done [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.060746] device-mapper: uevent: version 1.0.3 [ 4.063145] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.714892] virtio_net virtio0 ens2: renamed from eth0 [ 4.781190] scsi host0: ata_piix [ 4.793538] scsi host1: ata_piix [ 4.795579] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.797655] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.160232] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.644048] random: crng init done [ 9.645700] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.054892] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.154932] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.366220] SELinux: Disabled at runtime. [ 11.435753] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.441648] systemd[1]: Detected virtualization kvm. [ 11.442797] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.870201] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.873740] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.878945] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.884479] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.888188] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.897655] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.905460] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ 12.105515] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.317118] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.605315] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.686490] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.827865] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.845132] EDAC sbridge: Ver: 1.1.2 [ 13.974670] Key type dns_resolver registered [ 14.280590] NFS: Registering the id_resolver key type [ 14.282749] Key type id_resolver registered [ 14.284530] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg212-client login: [ 68.752942] libcfs: loading out-of-tree module taints kernel. [ 69.086462] Key type ._llcrypt registered [ 69.091953] Key type .llcrypt registered [ 70.120594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 70.140493] alg: No test for adler32 (adler32-zlib) [ 71.866612] Lustre: Lustre: Build Version: 2.17.57_1_g05abe65 [ 72.798435] LNet: Added LNI 192.168.202.12@tcp [8/256/0/180] [ 74.625172] Key type lgssc registered [ 76.477358] hrtimer: interrupt took 5344145 ns [ 77.357337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 213.147721] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 218.980493] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 234.456327] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing check_logdir /tmp/testlogs/ [ 238.562943] Lustre: lustre-OST0000-osc-ffff9a618548d000: disconnect after 22s idle [ 238.812350] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing yml_node [ 244.015044] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 246.437529] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 249.084453] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 250.835366] Lustre: DEBUG MARKER: -----============= acceptance-small: recovery-small ============----- Sat Aug 15 04:35:54 EDT 2026 [ 268.810473] Lustre: DEBUG MARKER: excepting tests: 136 [ 270.298355] Lustre: DEBUG MARKER: === recovery-small: start setup 04:36:14 (1786782974) === [ 274.928502] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing check_config_client /mnt/lustre [ 291.201302] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 301.664416] Lustre: DEBUG MARKER: === recovery-small: finish setup 04:36:45 (1786783005) === [ 303.532864] Lustre: DEBUG MARKER: == recovery-small test 1: create, chmod, stat: drop req, drop rep ========================================================== 04:36:47 (1786783007) [ 321.504242] Lustre: 10015:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786783010/real 1786783010] req@ffff9a6190880380 x1873577538236160/t0(0) o700->lustre-MDT0000-mdc-ffff9a618548d000@192.168.202.112@tcp:30/10 lens 264/248 e 0 to 1 dl 1786783026 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 321.522759] Lustre: lustre-MDT0000-mdc-ffff9a618548d000: Connection to lustre-MDT0000 (at 192.168.202.112@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 321.554681] Lustre: lustre-MDT0000-mdc-ffff9a618548d000: Connection restored to 192.168.202.112@tcp (at 192.168.202.112@tcp) [ 341.024365] Lustre: 10036:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786783030/real 1786783030] req@ffff9a61a1246680 x1873577538238208/t0(0) o36->lustre-MDT0000-mdc-ffff9a618548d000@192.168.202.112@tcp:12/10 lens 520/576 e 0 to 1 dl 1786783046 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 341.111879] Lustre: lustre-MDT0000-mdc-ffff9a618548d000: Connection to lustre-MDT0000 (at 192.168.202.112@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 341.194221] Lustre: lustre-MDT0000-mdc-ffff9a618548d000: Connection restored to 192.168.202.112@tcp (at 192.168.202.112@tcp) [ 361.440528] Lustre: 10062:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786783050/real 1786783050] req@ffff9a6190880000 x1873577538239488/t0(0) o101->lustre-MDT0000-mdc-ffff9a618548d000@192.168.202.112@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786783066 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'tchmod.0' uid:0 gid:0 projid:0 [ 361.463172] Lustre: lustre-MDT0000-mdc-ffff9a618548d000: Connection to lustre-MDT0000 (at 192.168.202.112@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 361.530875] Lustre: lustre-MDT0000-mdc-ffff9a618548d000: Connection restored to 192.168.202.112@tcp (at 192.168.202.112@tcp) [ 380.384208] Lustre: 10083:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786783069/real 1786783069] req@ffff9a6190ef0000 x1873577538241664/t0(0) o36->lustre-MDT0000-mdc-ffff9a618548d000@192.168.202.112@tcp:12/10 lens 488/512 e 0 to 1 dl 1786783085 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'tchmod.0' uid:0 gid:0 projid:4294967295 [ 380.414706] Lustre: lustre-MDT0000-mdc-ffff9a618548d000: Connection to lustre-MDT0000 (at 192.168.202.112@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 380.463859] Lustre: lustre-MDT0000-mdc-ffff9a618548d000: Connection restored to 192.168.202.112@tcp (at 192.168.202.112@tcp) [ 399.840270] Lustre: 10109:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786783089/real 1786783089] req@ffff9a6190ef0380 x1873577538242944/t0(0) o34->lustre-MDT0000-mdc-ffff9a618548d000@192.168.202.112@tcp:12/10 lens 472/728 e 0 to 1 dl 1786783105 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'statone.0' uid:0 gid:0 projid:0 [ 399.896629] Lustre: lustre-MDT0000-mdc-ffff9a618548d000: Connection to lustre-MDT0000 (at 192.168.202.112@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 399.969424] Lustre: lustre-MDT0000-mdc-ffff9a618548d000: Connection restored to 192.168.202.112@tcp (at 192.168.202.112@tcp) [ 419.297150] Lustre: 10130:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786783108/real 1786783108] req@ffff9a6190ef2a00 x1873577538244224/t0(0) o34->lustre-MDT0000-mdc-ffff9a618548d000@192.168.202.112@tcp:12/10 lens 472/728 e 0 to 1 dl 1786783124 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'statone.0' uid:0 gid:0 projid:0 [ 419.335312] Lustre: lustre-MDT0000-mdc-ffff9a618548d000: Connection to lustre-MDT0000 (at 192.168.202.112@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 419.376837] Lustre: lustre-MDT0000-mdc-ffff9a618548d000: Connection restored to 192.168.202.112@tcp (at 192.168.202.112@tcp) [ 428.164636] Lustre: DEBUG MARKER: == recovery-small test 4: open: drop req, drop rep ======= 04:38:52 (1786783132) [ 446.432168] Lustre: 10732:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786783135/real 1786783135] req@ffff9a61a1244a80 x1873577538246912/t0(0) o101->lustre-MDT0000-mdc-ffff9a618548d000@192.168.202.112@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786783151 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 [ 446.478184] Lustre: lustre-MDT0000-mdc-ffff9a618548d000: Connection to lustre-MDT0000 (at 192.168.202.112@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 446.519762] Lustre: lustre-MDT0000-mdc-ffff9a618548d000: Connection restored to 192.168.202.112@tcp (at 192.168.202.112@tcp) [ 473.868710] Lustre: DEBUG MARKER: == recovery-small test 5: rename: drop req, drop rep ===== 04:39:37 (1786783177) [ 491.488300] Lustre: 2352:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786783180/real 1786783180] req@ffff9a6190882a00 x1873577538253184/t0(0) o400->lustre-MDT0000-mdc-ffff9a618548d000@192.168.202.112@tcp:12/10 lens 224/224 e 0 to 1 dl 1786783196 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 491.516606] Lustre: 2352:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 493.539680] Lustre: lustre-MDT0000-mdc-ffff9a618548d000: Connection to lustre-MDT0000 (at 192.168.202.112@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 493.560466] Lustre: Skipped 1 previous similar message [ 493.622990] Lustre: lustre-MDT0000-mdc-ffff9a618548d000: Connection restored to 192.168.202.112@tcp (at 192.168.202.112@tcp) [ 493.634754] Lustre: Skipped 1 previous similar message [ 502.505973] Lustre: DEBUG MARKER: == recovery-small test 6: link, unlink: drop req, drop rep ========================================================== 04:40:06 (1786783206) [ 519.137040] Lustre: lustre-OST0001-osc-ffff9a618548d000: disconnect after 20s idle [ 519.150403] Lustre: Skipped 1 previous similar message [ 557.024207] Lustre: 12032:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786783246/real 1786783246] req@ffff9a6190882300 x1873577538263808/t0(0) o101->lustre-MDT0000-mdc-ffff9a618548d000@192.168.202.112@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786783262 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'unlink.0' uid:0 gid:0 projid:0 [ 557.041320] Lustre: 12032:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 576.482537] Lustre: lustre-MDT0000-mdc-ffff9a618548d000: Connection to lustre-MDT0000 (at 192.168.202.112@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 576.497673] Lustre: Skipped 3 previous similar messages [ 576.538772] Lustre: lustre-MDT0000-mdc-ffff9a618548d000: Connection restored to 192.168.202.112@tcp (at 192.168.202.112@tcp) [ 576.551961] Lustre: Skipped 3 previous similar messages [ 584.878359] Lustre: DEBUG MARKER: == recovery-small test 8: touch: drop rep (bug 1423) ===== 04:41:28 (1786783288) [ 612.265642] Lustre: DEBUG MARKER: == recovery-small test 9: pause bulk on OST (bug 1420) === 04:41:55 (1786783315) [ 629.087620] Lustre: DEBUG MARKER: == recovery-small test 10a: finish request on server after client eviction (bug 1521) ========================================================== 04:42:12 (1786783332) [ 629.658976] Lustre: *** cfs_fail_loc=305, val=0*** [ 631.936793] Lustre: *** cfs_fail_loc=305, val=0*** [ 643.553983] Lustre: lustre-OST0001-osc-ffff9a618548d000: disconnect after 21s idle [ 645.270760] Lustre: *** cfs_fail_loc=305, val=0*** [ 647.302570] Lustre: *** cfs_fail_loc=305, val=0*** [ 661.634322] Lustre: *** cfs_fail_loc=305, val=0*** [ 663.656935] Lustre: *** cfs_fail_loc=305, val=0*** [ 678.023785] Lustre: *** cfs_fail_loc=305, val=0*** [ 680.020381] Lustre: *** cfs_fail_loc=305, val=0*** [ 693.391165] Lustre: *** cfs_fail_loc=305, val=0*** [ 695.384904] Lustre: *** cfs_fail_loc=305, val=0*** [ 709.758199] Lustre: *** cfs_fail_loc=305, val=0*** [ 711.764596] Lustre: *** cfs_fail_loc=305, val=0*** [ 725.081610] Lustre: *** cfs_fail_loc=305, val=0*** [ 727.138343] Lustre: *** cfs_fail_loc=305, val=0*** [ 741.830105] LustreError: lustre-MDT0000-mdc-ffff9a618548d000: operation ldlm_enqueue to node 192.168.202.112@tcp failed: rc = -107 [ 741.847853] Lustre: lustre-MDT0000-mdc-ffff9a618548d000: Connection to lustre-MDT0000 (at 192.168.202.112@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 741.864622] Lustre: Skipped 1 previous similar message [ 741.874555] LustreError: lustre-MDT0000-mdc-ffff9a618548d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 741.904105] Lustre: lustre-MDT0000-mdc-ffff9a618548d000: Connection restored to 192.168.202.112@tcp (at 192.168.202.112@tcp) [ 741.923665] Lustre: Skipped 1 previous similar message [ 745.965596] Lustre: lustre-OST0000-osc-ffff9a618548d000: Connection to lustre-OST0000 (at 192.168.202.112@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 745.977876] LustreError: lustre-OST0000-osc-ffff9a618548d000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 745.985102] Lustre: lustre-OST0000-osc-ffff9a618548d000: Connection restored to 192.168.202.112@tcp (at 192.168.202.112@tcp) [ 752.320072] Lustre: DEBUG MARKER: == recovery-small test 10b: re-send BL AST =============== 04:44:15 (1786783455) [ 766.432331] Lustre: lustre-OST0000-osc-ffff9a618548d000: disconnect after 20s idle [ 777.352867] Lustre: DEBUG MARKER: == recovery-small test 10c: re-send BL AST vs reconnect race (LU-5569) ========================================================== 04:44:41 (1786783481) [ 784.690988] Lustre: DEBUG MARKER: == recovery-small test 10d: test failed blocking ast ===== 04:44:48 (1786783488) [ 788.187893] Lustre: Unmounted lustre-client [ 788.633314] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 790.282448] LustreError: lustre-OST0000-osc-ffff9a61836ff000: operation ldlm_enqueue to node 192.168.202.112@tcp failed: rc = -107 [ 790.301255] LustreError: lustre-OST0000-osc-ffff9a61836ff000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 790.312158] Lustre: 2351:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.112@tcp:/lustre/fid: [0x200000403:0x1:0x0]/ may get corrupted (rc -108) [ 793.103826] Lustre: Unmounted lustre-client [ 798.664660] Lustre: DEBUG MARKER: == recovery-small test 10e: re-send BL AST vs reconnect race 2 ========================================================== 04:45:02 (1786783502) [ 800.007639] Lustre: DEBUG MARKER: SKIP: recovery-small test_10e need two clients [ 801.356596] Lustre: DEBUG MARKER: == recovery-small test 11: wake up a thread waiting for completion after eviction (b=2460) ========================================================== 04:45:05 (1786783505) [ 830.081859] Lustre: DEBUG MARKER: == recovery-small test 12: recover from timed out resend in ptlrpcd (b=2494) ========================================================== 04:45:33 (1786783533) [ 830.288242] Lustre: DEBUG MARKER: multiop /mnt/lustre/f12.recovery-small OS_c [ 847.328245] Lustre: 17638:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786783536/real 1786783536] req@ffff9a61a1245f80 x1873577538335232/t0(0) o35->lustre-MDT0000-mdc-ffff9a61836ff000@192.168.202.112@tcp:23/10 lens 392/624 e 0 to 1 dl 1786783552 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 [ 847.361709] Lustre: 17638:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 873.757229] Lustre: DEBUG MARKER: == recovery-small test 13: mdc_readpage restart test (bug 1138) ========================================================== 04:46:17 (1786783577) [ 898.571053] Lustre: DEBUG MARKER: == recovery-small test 14: mdc_readpage resend test (bug 1138) ========================================================== 04:46:42 (1786783602) [ 908.417673] Lustre: DEBUG MARKER: == recovery-small test 15: failed open (-ENOMEM) ========= 04:46:52 (1786783612) [ 914.647527] Lustre: DEBUG MARKER: == recovery-small test 16: timeout bulk put, don't evict client (2732) ========================================================== 04:46:58 (1786783618) [ 962.510795] Lustre: DEBUG MARKER: == recovery-small test 17a: timeout bulk get, don't evict client (2732) ========================================================== 04:47:46 (1786783666) [ 1020.949196] Lustre: DEBUG MARKER: == recovery-small test 17b: timeout bulk get, dont evict client (3582) ========================================================== 04:48:43 (1786783723) [ 1022.317513] Lustre: DEBUG MARKER: SKIP: recovery-small test_17b Needs multiple clients [ 1024.809947] Lustre: DEBUG MARKER: == recovery-small test 18a: manual ost invalidate clears page cache immediately ========================================================== 04:48:48 (1786783728) [ 1025.507657] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1025.567350] LustreError: lustre-OST0001-osc-ffff9a61836ff000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1033.077588] Lustre: DEBUG MARKER: == recovery-small test 18b: eviction and reconnect clears page cache (2766) ========================================================== 04:48:56 (1786783736) [ 1040.370769] LustreError: lustre-OST0000-osc-ffff9a61836ff000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1064.493348] Lustre: DEBUG MARKER: == recovery-small test 18c: Dropped connect reply after eviction handing (14755) ========================================================== 04:49:28 (1786783768) [ 1067.950949] LustreError: lustre-OST0000-osc-ffff9a61836ff000: operation ost_statfs to node 192.168.202.112@tcp failed: rc = -107 [ 1078.261560] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.202.112@tcp) after server handle changed from 0xb1b3a8da059a5052 to 0xb1b3a8da059a51a2 [ 1078.287198] LustreError: lustre-OST0000-osc-ffff9a61836ff000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1087.179230] Lustre: DEBUG MARKER: == recovery-small test 19a: test expired_lock_main on mds (2867) ========================================================== 04:49:51 (1786783791) [ 1088.003147] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 1088.008224] Lustre: Skipped 1 previous similar message [ 1105.380321] Lustre: 2351:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786783794/real 1786783794] req@ffff9a61a1246d80 x1873577538392704/t0(0) o103->lustre-MDT0000-mdc-ffff9a61836ff000@192.168.202.112@tcp:17/18 lens 328/224 e 0 to 1 dl 1786783810 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 [ 1105.405300] Lustre: 2351:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1194.970656] LustreError: lustre-MDT0000-mdc-ffff9a61836ff000: operation ldlm_enqueue to node 192.168.202.112@tcp failed: rc = -107 [ 1194.985514] LustreError: lustre-MDT0000-mdc-ffff9a61836ff000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1194.995389] LustreError: 23886:0:(file.c:6156:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5 [ 1195.794465] Lustre: Unmounted lustre-client [ 1203.484699] Lustre: DEBUG MARKER: == recovery-small test 19b: test expired_lock_main on ost (2867) ========================================================== 04:51:47 (1786783907) [ 1203.971033] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 1268.704416] Lustre: lustre-OST0000-osc-ffff9a61836ff000: Connection to lustre-OST0000 (at 192.168.202.112@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1268.738664] Lustre: Skipped 18 previous similar messages [ 1268.756577] Lustre: lustre-OST0000-osc-ffff9a61836ff000: Connection restored to 192.168.202.112@tcp (at 192.168.202.112@tcp) [ 1268.771403] Lustre: Skipped 15 previous similar messages [ 1311.739963] Lustre: Unmounted lustre-client [ 1311.763399] LustreError: lustre-OST0000-osc-ffff9a61836ff000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1318.924413] Lustre: DEBUG MARKER: == recovery-small test 19c: check reconnect and lock resend do not trigger expired_lock_main ========================================================== 04:53:42 (1786784022) [ 1319.340252] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 1324.700423] Lustre: Unmounted lustre-client [ 1335.901974] Lustre: DEBUG MARKER: == recovery-small test 20a: ldlm_handle_enqueue error (should return error) ========================================================== 04:54:00 (1786784040) [ 1337.828557] LustreError: lustre-OST0000-osc-ffff9a61836ff000: operation ldlm_enqueue to node 192.168.202.112@tcp failed: rc = -12 [ 1345.925351] Lustre: DEBUG MARKER: == recovery-small test 20b: ldlm_handle_enqueue error (should return error) ========================================================== 04:54:09 (1786784049) [ 1347.488676] LustreError: lustre-OST0000-osc-ffff9a61836ff000: operation ldlm_enqueue to node 192.168.202.112@tcp failed: rc = -12 [ 1355.247304] Lustre: DEBUG MARKER: == recovery-small test 21a: drop close request while close and open are both in flight ========================================================== 04:54:19 (1786784059) [ 1388.065370] Lustre: DEBUG MARKER: == recovery-small test 21b: drop open request while close and open are both in flight ========================================================== 04:54:50 (1786784090) [ 1537.930761] Lustre: DEBUG MARKER: == recovery-small test 21c: drop both request while close and open are both in flight ========================================================== 04:57:22 (1786784242) [ 1570.017714] Lustre: DEBUG MARKER: == recovery-small test 21d: drop close reply while close and open are both in flight ========================================================== 04:57:54 (1786784274) [ 1598.276469] Lustre: DEBUG MARKER: == recovery-small test 21e: drop open reply while close and open are both in flight ========================================================== 04:58:22 (1786784302) [ 1738.721179] Lustre: 29640:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786784304/real 1786784304] req@ffff9a6190883b80 x1873577538502784/t0(0) o36->lustre-MDT0000-mdc-ffff9a61836ff000@192.168.202.112@tcp:12/10 lens 488/512 e 0 to 1 dl 1786784444 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 1738.757710] Lustre: 29640:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [ 1740.758362] Lustre: DEBUG MARKER: == recovery-small test 21f: drop both reply while close and open are both in flight ========================================================== 05:00:44 (1786784444) [ 1768.197509] Lustre: DEBUG MARKER: == recovery-small test 21g: drop open reply and close request while close and open are both in flight ========================================================== 05:01:12 (1786784472) [ 1794.478686] Lustre: DEBUG MARKER: == recovery-small test 21h: drop open request and close reply while close and open are both in flight ========================================================== 05:01:38 (1786784498) [ 1822.030270] Lustre: DEBUG MARKER: == recovery-small test 22: drop close request and do mknod ========================================================== 05:02:05 (1786784525) [ 1847.220633] Lustre: DEBUG MARKER: == recovery-small test 23: client hang when close a file after mds crash ========================================================== 05:02:31 (1786784551) [ 1870.304784] Lustre: lustre-MDT0000-mdc-ffff9a61836ff000: Connection to lustre-MDT0000 (at 192.168.202.112@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1870.317370] Lustre: Skipped 12 previous similar messages [ 1876.449808] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 192.168.202.112@tcp) was lost; in progress operations using this service will fail [ 1876.479691] Lustre: Evicted from MGS (at 192.168.202.112@tcp) after server handle changed from 0xb1b3a8da059a4730 to 0xb1b3a8da059a6bf7 [ 1876.493339] Lustre: MGC192.168.202.112@tcp: Connection restored to 192.168.202.112@tcp (at 192.168.202.112@tcp) [ 1876.516550] Lustre: Skipped 12 previous similar messages [ 1878.255883] Lustre: 15919:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.112@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 1900.618623] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1902.354286] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1909.452134] Lustre: DEBUG MARKER: == recovery-small test 24a: fsync error (should return error) ========================================================== 05:03:33 (1786784613) [ 1910.954544] LustreError: lustre-OST0000-osc-ffff9a61836ff000: operation ost_write to node 192.168.202.112@tcp failed: rc = -107 [ 1910.987703] LustreError: lustre-OST0000-osc-ffff9a61836ff000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1911.003127] Lustre: 2349:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.112@tcp:/lustre/fid: [0x200000404:0x30:0x0]// may get corrupted (rc -5) [ 1918.769645] Lustre: DEBUG MARKER: == recovery-small test 24b: test dirty page discard due to client eviction ========================================================== 05:03:42 (1786784622) [ 1920.500900] LustreError: lustre-OST0000-osc-ffff9a61836ff000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1920.513914] Lustre: 2352:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.202.112@tcp:/lustre/fid: [0x200000404:0x35:0x0]// may get corrupted (rc -108) [ 1922.559952] Lustre: DEBUG MARKER: recovery-small test_24b: @@@@@@ IGNORE (bz5494): multiop didn't fail fsync: 0 or close: 5 [ 1930.339047] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 05:03:54 (1786784634) [ 1932.451269] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node [ 1934.082832] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 05:03:58 (1786784638) [ 1936.148450] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node [ 1937.674204] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 05:04:01 (1786784641) [ 1961.460213] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 192.168.202.112@tcp) was lost; in progress operations using this service will fail [ 1961.483664] Lustre: Evicted from MGS (at 192.168.202.112@tcp) after server handle changed from 0xb1b3a8da059a6bf7 to 0xb1b3a8da059a858f [ 1963.435053] Lustre: 15919:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.112@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 1980.944485] BUG: spinlock recursion on CPU#0, ptlrpcd_rcv/2348 [ 1980.957335] lock: 0xffff9a61907912e0, .magic: dead4ead, .owner: ptlrpcd_rcv/2348, .owner_cpu: 0 [ 1980.982884] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1980.994695] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1981.007592] Call Trace: [ 1981.009071] ? dump_stack+0xbb/0x10e [ 1981.013885] ? spin_dump.cold.3+0x32/0x65 [ 1981.018275] ? do_raw_spin_lock+0x10f/0x190 [ 1981.028379] ? _raw_spin_lock+0x21/0x30 [ 1981.032696] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 1981.040326] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 1981.047015] ? mdc_commit_open+0x91/0xb0 [mdc] [ 1981.054474] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 1981.058363] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 1981.062123] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 1981.065488] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 1981.072445] ? __next_timer_interrupt+0x160/0x160 [ 1981.079504] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 1981.084471] ? do_wait_intr+0xf0/0xf0 [ 1981.091459] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 1981.100764] ? kthread+0x1d1/0x200 [ 1981.110666] ? set_kthread_struct+0x70/0x70 [ 1981.115563] ? ret_from_fork+0x1f/0x30 [ 2008.058413] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 2008.060834] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 2008.066466] Modules linked in: [ 2008.071542] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 2008.071556] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2008.071638] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 2008.071643] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2008.071646] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2008.071661] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2008.071665] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2008.071669] RAX: 0000000000007f88 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2008.071672] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2008.071674] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2008.071677] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2008.071679] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2008.071682] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2008.071685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2008.071688] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2008.071697] Call Trace: [ 2008.071702] [ 2008.071707] ? show_regs.cold.9+0x22/0x2f [ 2008.071716] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2008.071733] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2008.071738] ? watchdog+0x40/0x40 [ 2008.071742] ? hrtimer_interrupt+0x128/0x2c0 [ 2008.071747] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2008.071758] ? apic_timer_interrupt+0xf/0x20 [ 2008.071763] [ 2008.071766] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2008.071771] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2008.071776] do_raw_spin_lock+0x177/0x190 [ 2008.071779] Modules linked in: [ 2008.071782] _raw_spin_lock+0x21/0x30 [ 2008.071784] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2008.071844] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 2008.071850] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2008.071792] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2008.072140] ? do_raw_spin_unlock+0x75/0x190 [ 2008.072146] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2008.072363] ? do_wait_intr+0xf0/0xf0 [ 2008.072371] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2008.073075] mdc_reint+0x2d/0x110 [mdc] [ 2008.073117] mdc_unlink+0x658/0x870 [mdc] [ 2008.073143] lmv_unlink+0x34d/0xbd0 [lmv] [ 2008.073170] ll_unlink+0x24d/0x7f0 [lustre] [ 2008.073278] vfs_unlink+0x202/0x330 [ 2008.073286] do_unlinkat+0x2fa/0x460 [ 2008.073291] __x64_sys_unlink+0x37/0x50 [ 2008.073294] do_syscall_64+0xc1/0x440 [ 2008.073301] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2008.073308] RIP: 0033:0x7f83e5f5c10b [ 2008.073313] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2008.073316] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2008.073321] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2008.073323] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2008.073325] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2008.073327] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2008.073328] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2008.071854] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2008.073665] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2008.073674] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2008.073677] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2008.073682] RAX: 0000000000007e9e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2008.073686] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2008.073689] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2008.073691] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2008.073693] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2008.073697] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2008.073699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2008.073701] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2008.073709] Call Trace: [ 2008.073713] [ 2008.073716] ? show_regs.cold.9+0x22/0x2f [ 2008.073726] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2008.073732] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2008.073737] ? watchdog+0x40/0x40 [ 2008.073741] ? hrtimer_interrupt+0x128/0x2c0 [ 2008.073745] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2008.073752] ? apic_timer_interrupt+0xf/0x20 [ 2008.073756] [ 2008.073759] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2008.073764] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2008.073769] do_raw_spin_lock+0x177/0x190 [ 2008.073774] _raw_spin_lock+0x21/0x30 [ 2008.073781] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2008.074657] process_one_work+0x2c8/0x700 [ 2008.074666] worker_thread+0x296/0x6e0 [ 2008.074670] ? rescuer_thread+0x570/0x570 [ 2008.074676] kthread+0x1d1/0x200 [ 2008.074681] ? set_kthread_struct+0x70/0x70 [ 2008.074685] ret_from_fork+0x1f/0x30 [ 2008.705477] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2008.767173] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2008.792936] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2008.808897] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2008.824481] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2008.845355] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2008.850735] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2008.854514] RDX: 0000000000006cf9 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2008.857777] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2008.861520] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2008.863893] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2008.866343] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2008.869960] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2008.873014] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2008.876195] Call Trace: [ 2008.877340] [ 2008.878350] ? show_regs.cold.9+0x22/0x2f [ 2008.880653] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2008.882776] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2008.884780] ? watchdog+0x40/0x40 [ 2008.886225] ? hrtimer_interrupt+0x128/0x2c0 [ 2008.891916] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2008.893578] ? apic_timer_interrupt+0xf/0x20 [ 2008.895099] [ 2008.900286] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2008.902988] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2008.910433] do_raw_spin_lock+0x177/0x190 [ 2008.916072] _raw_spin_lock+0x21/0x30 [ 2008.920062] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2008.927427] obd_mod_free+0x46/0x140 [ptlrpc] [ 2008.931954] mdc_commit_open+0x91/0xb0 [mdc] [ 2008.935449] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2008.938547] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2008.943884] after_reply+0xe2c/0x1780 [ptlrpc] [ 2008.949288] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2008.954450] ? __next_timer_interrupt+0x160/0x160 [ 2008.958500] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2008.962428] ? do_wait_intr+0xf0/0xf0 [ 2008.964582] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2008.966935] kthread+0x1d1/0x200 [ 2008.968458] ? set_kthread_struct+0x70/0x70 [ 2008.974292] ret_from_fork+0x1f/0x30 [ 2036.056132] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 2036.060234] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2036.061381] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 2036.066555] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 2036.066568] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2036.066918] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2036.066983] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2036.066986] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2036.067176] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2036.067182] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2036.067186] RAX: 00000000000006ff RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2036.067189] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2036.067191] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2036.067194] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2036.067197] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2036.067208] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2036.067212] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2036.067215] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2036.067223] Call Trace: [ 2036.067229] [ 2036.067234] ? show_regs.cold.9+0x22/0x2f [ 2036.067245] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2036.067251] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2036.067256] ? watchdog+0x40/0x40 [ 2036.067260] ? hrtimer_interrupt+0x128/0x2c0 [ 2036.067264] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2036.067273] ? apic_timer_interrupt+0xf/0x20 [ 2036.067279] [ 2036.067282] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2036.067287] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2036.067292] do_raw_spin_lock+0x177/0x190 [ 2036.067296] _raw_spin_lock+0x21/0x30 [ 2036.067303] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2036.068281] ? do_raw_spin_unlock+0x75/0x190 [ 2036.068288] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2036.069302] ? do_wait_intr+0xf0/0xf0 [ 2036.069313] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2036.070935] mdc_reint+0x2d/0x110 [mdc] [ 2036.070974] mdc_unlink+0x658/0x870 [mdc] [ 2036.071499] lmv_unlink+0x34d/0xbd0 [lmv] [ 2036.071530] ll_unlink+0x24d/0x7f0 [lustre] [ 2036.071928] vfs_unlink+0x202/0x330 [ 2036.071942] do_unlinkat+0x2fa/0x460 [ 2036.071947] __x64_sys_unlink+0x37/0x50 [ 2036.071951] do_syscall_64+0xc1/0x440 [ 2036.071959] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2036.071970] RIP: 0033:0x7f83e5f5c10b [ 2036.071978] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2036.071982] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2036.071988] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2036.071991] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2036.071993] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2036.071995] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2036.072007] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2036.118262] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2036.118278] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2036.118282] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2036.118300] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2036.118305] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2036.118310] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2036.118313] RDX: 000000000000059b RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2036.118315] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2036.118317] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2036.118319] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2036.118322] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2036.118324] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2036.118327] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2036.118335] Call Trace: [ 2036.118340] [ 2036.118346] ? show_regs.cold.9+0x22/0x2f [ 2036.118360] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2036.118367] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2036.118372] ? watchdog+0x40/0x40 [ 2036.118377] ? hrtimer_interrupt+0x128/0x2c0 [ 2036.118381] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2036.118391] ? apic_timer_interrupt+0xf/0x20 [ 2036.118395] [ 2036.118399] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2036.118404] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2036.118409] do_raw_spin_lock+0x177/0x190 [ 2036.118414] _raw_spin_lock+0x21/0x30 [ 2036.118422] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2036.121235] obd_mod_free+0x46/0x140 [ptlrpc] [ 2036.121507] mdc_commit_open+0x91/0xb0 [mdc] [ 2036.121558] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2036.122345] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2036.130651] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2036.130735] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2036.130746] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2036.130752] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2036.131066] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2036.131078] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2036.131083] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2036.131087] RAX: 0000000000001e15 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2036.131089] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2036.131091] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2036.131093] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2036.131096] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2036.131099] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2036.131102] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2036.131104] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2036.131112] Call Trace: [ 2036.131118] [ 2036.131122] ? show_regs.cold.9+0x22/0x2f [ 2036.131133] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2036.131141] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2036.131145] ? watchdog+0x40/0x40 [ 2036.131149] ? hrtimer_interrupt+0x128/0x2c0 [ 2036.131153] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2036.131163] ? apic_timer_interrupt+0xf/0x20 [ 2036.131168] [ 2036.131170] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2036.131175] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2036.131180] do_raw_spin_lock+0x177/0x190 [ 2036.131184] _raw_spin_lock+0x21/0x30 [ 2036.131188] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2036.131412] process_one_work+0x2c8/0x700 [ 2036.131420] worker_thread+0x296/0x6e0 [ 2036.131424] ? rescuer_thread+0x570/0x570 [ 2036.131427] kthread+0x1d1/0x200 [ 2036.131431] ? set_kthread_struct+0x70/0x70 [ 2036.131435] ret_from_fork+0x1f/0x30 [ 2036.685832] after_reply+0xe2c/0x1780 [ptlrpc] [ 2036.687885] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2036.690233] ? __next_timer_interrupt+0x160/0x160 [ 2036.692441] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2036.694605] ? do_wait_intr+0xf0/0xf0 [ 2036.695788] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2036.697556] kthread+0x1d1/0x200 [ 2036.705445] ? set_kthread_struct+0x70/0x70 [ 2036.708523] ret_from_fork+0x1f/0x30 [ 2040.961035] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 2040.968529] rcu: 0-....: (37929 ticks this GP) idle=30a/1/0x4000000000000000 softirq=128519/128519 fqs=13862 [ 2040.978924] (detected by 1, t=60002 jiffies, g=262441, q=26493) [ 2040.982432] Sending NMI from CPU 1 to CPUs 0: [ 2040.986074] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 1.475 msecs [ 2040.986585] NMI backtrace for cpu 0 [ 2040.986588] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2040.986590] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2040.986592] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2040.986593] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2040.986595] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 [ 2040.986598] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2040.986599] RDX: 0000000000006936 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2040.986600] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2040.986601] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2040.986602] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2040.986603] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2040.986604] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2040.986606] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2040.986607] Call Trace: [ 2040.986607] [ 2040.986608] ? show_regs.cold.9+0x22/0x2f [ 2040.986609] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2040.986610] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2040.986611] ? nmi_handle+0xbd/0x210 [ 2040.986612] ? default_do_nmi+0x4b/0x1b0 [ 2040.986613] ? do_nmi+0x280/0x380 [ 2040.986614] ? end_repeat_nmi+0x16/0x1f [ 2040.986615] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2040.986616] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2040.986617] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2040.986617] [ 2040.986618] do_raw_spin_lock+0x177/0x190 [ 2040.986619] _raw_spin_lock+0x21/0x30 [ 2040.986620] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2040.986621] obd_mod_free+0x46/0x140 [ptlrpc] [ 2040.986622] mdc_commit_open+0x91/0xb0 [mdc] [ 2040.986624] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2040.986625] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2040.986626] after_reply+0xe2c/0x1780 [ptlrpc] [ 2040.986627] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2040.986628] ? __next_timer_interrupt+0x160/0x160 [ 2040.986629] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2040.986630] ? do_wait_intr+0xf0/0xf0 [ 2040.986630] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2040.986632] kthread+0x1d1/0x200 [ 2040.986632] ? set_kthread_struct+0x70/0x70 [ 2040.986633] ret_from_fork+0x1f/0x30 [ 2064.060061] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 2064.062087] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 2064.064849] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2064.065118] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2064.065125] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2064.065137] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2064.065558] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2064.073392] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2064.073473] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2064.073480] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2064.073483] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2064.073503] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2064.073507] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2064.073512] RAX: 0000000000005805 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2064.073514] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2064.073516] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2064.073517] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2064.073519] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2064.073522] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2064.073524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2064.073526] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2064.073535] Call Trace: [ 2064.073541] [ 2064.073547] ? show_regs.cold.9+0x22/0x2f [ 2064.073558] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2064.073566] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2064.073570] ? watchdog+0x40/0x40 [ 2064.073575] ? hrtimer_interrupt+0x128/0x2c0 [ 2064.073579] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2064.073591] ? apic_timer_interrupt+0xf/0x20 [ 2064.073594] [ 2064.073597] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2064.073603] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2064.134396] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2064.134409] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2064.134416] RAX: 0000000000005b1c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2064.134418] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2064.134420] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2064.134423] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2064.134425] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2064.134428] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2064.134431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2064.134434] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2064.134442] Call Trace: [ 2064.134447] [ 2064.134454] ? show_regs.cold.9+0x22/0x2f [ 2064.134468] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2064.134476] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2064.134480] ? watchdog+0x40/0x40 [ 2064.134484] ? hrtimer_interrupt+0x128/0x2c0 [ 2064.134488] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2064.134497] ? apic_timer_interrupt+0xf/0x20 [ 2064.134502] [ 2064.134505] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2064.134514] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2064.134520] do_raw_spin_lock+0x177/0x190 [ 2064.134525] _raw_spin_lock+0x21/0x30 [ 2064.134530] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2064.140101] do_raw_spin_lock+0x177/0x190 [ 2064.140122] _raw_spin_lock+0x21/0x30 [ 2064.140183] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2064.140463] ? do_raw_spin_unlock+0x75/0x190 [ 2064.140468] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2064.140926] ? do_wait_intr+0xf0/0xf0 [ 2064.140946] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2064.141241] mdc_reint+0x2d/0x110 [mdc] [ 2064.141278] mdc_unlink+0x658/0x870 [mdc] [ 2064.141306] lmv_unlink+0x34d/0xbd0 [lmv] [ 2064.141338] ll_unlink+0x24d/0x7f0 [lustre] [ 2064.141455] vfs_unlink+0x202/0x330 [ 2064.141467] do_unlinkat+0x2fa/0x460 [ 2064.141471] __x64_sys_unlink+0x37/0x50 [ 2064.141474] do_syscall_64+0xc1/0x440 [ 2064.141483] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2064.141491] RIP: 0033:0x7f83e5f5c10b [ 2064.141499] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2064.141503] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2064.141508] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2064.141510] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2064.141512] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2064.141514] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2064.141516] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2064.484941] process_one_work+0x2c8/0x700 [ 2064.487777] worker_thread+0x296/0x6e0 [ 2064.489531] ? rescuer_thread+0x570/0x570 [ 2064.491351] kthread+0x1d1/0x200 [ 2064.492490] ? set_kthread_struct+0x70/0x70 [ 2064.494386] ret_from_fork+0x1f/0x30 [ 2068.057101] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 2068.067252] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2068.101500] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2068.107700] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2068.115485] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2068.121126] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2068.138640] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2068.143187] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2068.150156] RDX: 0000000000005427 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2068.157672] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2068.161889] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2068.165775] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2068.171457] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2068.177375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2068.183286] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2068.188820] Call Trace: [ 2068.190518] [ 2068.191664] ? show_regs.cold.9+0x22/0x2f [ 2068.193518] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2068.196497] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2068.198523] ? watchdog+0x40/0x40 [ 2068.200683] ? hrtimer_interrupt+0x128/0x2c0 [ 2068.207902] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2068.210227] ? apic_timer_interrupt+0xf/0x20 [ 2068.213112] [ 2068.214061] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2068.217608] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2068.220310] do_raw_spin_lock+0x177/0x190 [ 2068.222530] _raw_spin_lock+0x21/0x30 [ 2068.224501] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2068.227439] obd_mod_free+0x46/0x140 [ptlrpc] [ 2068.230445] mdc_commit_open+0x91/0xb0 [mdc] [ 2068.232741] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2068.236886] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2068.239162] after_reply+0xe2c/0x1780 [ptlrpc] [ 2068.241235] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2068.248511] ? __next_timer_interrupt+0x160/0x160 [ 2068.250779] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2068.253460] ? do_wait_intr+0xf0/0xf0 [ 2068.255944] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2068.258973] kthread+0x1d1/0x200 [ 2068.263203] ? set_kthread_struct+0x70/0x70 [ 2068.265057] ret_from_fork+0x1f/0x30 [ 2090.102133] INFO: task writemany:36353 blocked for more than 120 seconds. [ 2090.104864] Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2090.108665] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2090.112339] task:writemany state:D stack:0 pid:36353 ppid:36350 flags:0x80000000 [ 2090.115803] Call Trace: [ 2090.117095] __schedule+0x351/0xcb0 [ 2090.118961] schedule+0xc0/0x180 [ 2090.120332] schedule_preempt_disabled+0x21/0x40 [ 2090.122607] rwsem_down_write_slowpath+0x5d7/0xa40 [ 2090.125097] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2090.128394] down_write+0x80/0xd0 [ 2090.130083] do_last+0x2eb/0xfc0 [ 2090.131426] ? nd_jump_root+0xe5/0x160 [ 2090.133558] ? path_init+0x437/0x520 [ 2090.135236] path_openat+0xf7/0x500 [ 2090.137358] ? do_raw_spin_unlock+0x75/0x190 [ 2090.140013] do_filp_open+0x99/0x140 [ 2090.152852] ? getname_flags+0x6e/0x330 [ 2090.156514] ? __check_object_size+0xff/0x256 [ 2090.165052] ? do_raw_spin_unlock+0x75/0x190 [ 2090.169557] ? _raw_spin_unlock+0x12/0x30 [ 2090.174296] do_sys_openat2+0x2b4/0x410 [ 2090.177491] do_sys_open+0x73/0xa0 [ 2090.180129] __x64_sys_openat+0x24/0x30 [ 2090.185280] do_syscall_64+0xc1/0x440 [ 2090.188316] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2090.190336] RIP: 0033:0x7f83e6051332 [ 2090.193583] Code: Unable to access opcode bytes at RIP 0x7f83e6051308. [ 2090.197977] RSP: 002b:00007fff7d2bcf90 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2090.202215] RAX: ffffffffffffffda RBX: 0000000000001538 RCX: 00007f83e6051332 [ 2090.205018] RDX: 0000000000000042 RSI: 00007fff7d2bd060 RDI: 00000000ffffff9c [ 2090.208586] RBP: 00007fff7d2bd460 R08: 0000000000000000 R09: 0000000000000000 [ 2090.211822] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff7d2bd060 [ 2090.214952] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2090.224850] INFO: task writemany:36354 blocked for more than 120 seconds. [ 2090.227396] Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2090.230372] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2090.233730] task:writemany state:D stack:0 pid:36354 ppid:36350 flags:0x80000000 [ 2090.236620] Call Trace: [ 2090.237500] __schedule+0x351/0xcb0 [ 2090.238952] schedule+0xc0/0x180 [ 2090.240155] schedule_preempt_disabled+0x21/0x40 [ 2090.241933] rwsem_down_write_slowpath+0x5d7/0xa40 [ 2090.244117] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2090.246360] down_write+0x80/0xd0 [ 2090.247526] do_last+0x2eb/0xfc0 [ 2090.248512] ? nd_jump_root+0xe5/0x160 [ 2090.249934] ? path_init+0x437/0x520 [ 2090.251123] path_openat+0xf7/0x500 [ 2090.252179] ? do_raw_spin_unlock+0x75/0x190 [ 2090.254015] do_filp_open+0x99/0x140 [ 2090.255164] ? getname_flags+0x6e/0x330 [ 2090.256325] ? __check_object_size+0xff/0x256 [ 2090.257822] ? do_raw_spin_unlock+0x75/0x190 [ 2090.259317] ? _raw_spin_unlock+0x12/0x30 [ 2090.261026] do_sys_openat2+0x2b4/0x410 [ 2090.262214] do_sys_open+0x73/0xa0 [ 2090.263306] __x64_sys_openat+0x24/0x30 [ 2090.264513] do_syscall_64+0xc1/0x440 [ 2090.265838] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2090.267455] RIP: 0033:0x7f83e6051332 [ 2090.268587] Code: Unable to access opcode bytes at RIP 0x7f83e6051308. [ 2090.270611] RSP: 002b:00007fff7d2bcf90 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2090.273032] RAX: ffffffffffffffda RBX: 000000000000a058 RCX: 00007f83e6051332 [ 2090.275208] RDX: 0000000000000042 RSI: 00007fff7d2bd060 RDI: 00000000ffffff9c [ 2090.277403] RBP: 00007fff7d2bd460 R08: 0000000000000000 R09: 0000000000000000 [ 2090.279523] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff7d2bd060 [ 2090.282157] R13: 00000000ffffffff R14: 000000000000000f R15: 0000000000000000 [ 2090.284975] INFO: task writemany:36355 blocked for more than 120 seconds. [ 2090.287306] Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2090.290126] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2090.296049] task:writemany state:D stack:0 pid:36355 ppid:36350 flags:0x80000000 [ 2090.299534] Call Trace: [ 2090.300721] __schedule+0x351/0xcb0 [ 2090.302308] schedule+0xc0/0x180 [ 2090.303893] schedule_preempt_disabled+0x21/0x40 [ 2090.305965] rwsem_down_write_slowpath+0x5d7/0xa40 [ 2090.307687] ? __might_sleep+0x59/0xc0 [ 2090.309644] down_write+0x80/0xd0 [ 2090.310847] do_unlinkat+0x184/0x460 [ 2090.312430] __x64_sys_unlink+0x37/0x50 [ 2090.313966] do_syscall_64+0xc1/0x440 [ 2090.315361] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2090.317166] RIP: 0033:0x7f83e5f5c10b [ 2090.318506] Code: Unable to access opcode bytes at RIP 0x7f83e5f5c0e1. [ 2090.320940] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2090.323440] RAX: ffffffffffffffda RBX: 000000000002c258 RCX: 00007f83e5f5c10b [ 2090.325512] RDX: 0000000000000038 RSI: 00007fff7d2bd460 RDI: 00007fff7d2bd060 [ 2090.327562] RBP: 00007fff7d2bd460 R08: 00007f83e62f421c R09: 00007f83e62f4260 [ 2090.329683] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2090.332099] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2090.336633] INFO: task writemany:36356 blocked for more than 120 seconds. [ 2090.338828] Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2090.341212] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2090.343594] task:writemany state:D stack:0 pid:36356 ppid:36350 flags:0x80000000 [ 2090.346589] Call Trace: [ 2090.347478] __schedule+0x351/0xcb0 [ 2090.348585] schedule+0xc0/0x180 [ 2090.349606] schedule_preempt_disabled+0x21/0x40 [ 2090.351211] rwsem_down_write_slowpath+0x5d7/0xa40 [ 2090.352580] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2090.354715] down_write+0x80/0xd0 [ 2090.355625] do_last+0x2eb/0xfc0 [ 2090.357057] ? nd_jump_root+0xe5/0x160 [ 2090.358228] ? path_init+0x437/0x520 [ 2090.359359] path_openat+0xf7/0x500 [ 2090.360608] ? do_raw_spin_unlock+0x75/0x190 [ 2090.362103] do_filp_open+0x99/0x140 [ 2090.363161] ? getname_flags+0x6e/0x330 [ 2090.364378] ? __check_object_size+0xff/0x256 [ 2090.365931] ? do_raw_spin_unlock+0x75/0x190 [ 2090.367211] ? _raw_spin_unlock+0x12/0x30 [ 2090.368615] do_sys_openat2+0x2b4/0x410 [ 2090.369932] do_sys_open+0x73/0xa0 [ 2090.370926] __x64_sys_openat+0x24/0x30 [ 2090.372065] do_syscall_64+0xc1/0x440 [ 2090.373333] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2090.374852] RIP: 0033:0x7f83e6051332 [ 2090.376101] Code: Unable to access opcode bytes at RIP 0x7f83e6051308. [ 2090.378065] RSP: 002b:00007fff7d2bcf90 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2090.380409] RAX: ffffffffffffffda RBX: 000000000008ebc0 RCX: 00007f83e6051332 [ 2090.382405] RDX: 0000000000000042 RSI: 00007fff7d2bd060 RDI: 00000000ffffff9c [ 2090.385108] RBP: 00007fff7d2bd460 R08: 0000000000000000 R09: 0000000000000000 [ 2090.392762] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff7d2bd060 [ 2090.397989] R13: 00000000ffffffff R14: 0000000000000002 R15: 0000000000000000 [ 2092.060015] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 2092.063273] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2092.065497] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2092.088717] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2092.088730] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2092.088850] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2092.089134] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2092.089147] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2092.089152] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2092.089156] RAX: 00000000000061b9 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2092.089159] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2092.089161] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2092.089163] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2092.089166] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2092.089270] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2092.089273] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2092.089276] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2092.089283] Call Trace: [ 2092.089288] [ 2092.089294] ? show_regs.cold.9+0x22/0x2f [ 2092.089305] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2092.089312] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2092.089317] ? watchdog+0x40/0x40 [ 2092.089321] ? hrtimer_interrupt+0x128/0x2c0 [ 2092.089325] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2092.089337] ? apic_timer_interrupt+0xf/0x20 [ 2092.089342] [ 2092.089345] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2092.089349] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2092.089354] do_raw_spin_lock+0x177/0x190 [ 2092.089366] _raw_spin_lock+0x21/0x30 [ 2092.089371] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2092.089681] process_one_work+0x2c8/0x700 [ 2092.089690] worker_thread+0x296/0x6e0 [ 2092.089695] ? rescuer_thread+0x570/0x570 [ 2092.089699] kthread+0x1d1/0x200 [ 2092.089704] ? set_kthread_struct+0x70/0x70 [ 2092.089708] ret_from_fork+0x1f/0x30 [ 2092.191361] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2092.212717] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2092.217435] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2092.220745] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2092.223526] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2092.232174] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2092.234926] RAX: 00000000000054f7 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2092.238373] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2092.241804] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2092.245649] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2092.249611] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2092.253682] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2092.258244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2092.260909] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2092.265489] Call Trace: [ 2092.266637] [ 2092.267768] ? show_regs.cold.9+0x22/0x2f [ 2092.270381] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2092.272589] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2092.275049] ? watchdog+0x40/0x40 [ 2092.276912] ? hrtimer_interrupt+0x128/0x2c0 [ 2092.279755] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2092.281990] ? apic_timer_interrupt+0xf/0x20 [ 2092.284541] [ 2092.285514] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2092.288278] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2092.290844] do_raw_spin_lock+0x177/0x190 [ 2092.292535] _raw_spin_lock+0x21/0x30 [ 2092.294574] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2092.297646] ? do_raw_spin_unlock+0x75/0x190 [ 2092.299807] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2092.302747] ? do_wait_intr+0xf0/0xf0 [ 2092.304468] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2092.307426] mdc_reint+0x2d/0x110 [mdc] [ 2092.310019] mdc_unlink+0x658/0x870 [mdc] [ 2092.311960] lmv_unlink+0x34d/0xbd0 [lmv] [ 2092.314229] ll_unlink+0x24d/0x7f0 [lustre] [ 2092.315814] vfs_unlink+0x202/0x330 [ 2092.317184] do_unlinkat+0x2fa/0x460 [ 2092.319344] __x64_sys_unlink+0x37/0x50 [ 2092.321300] do_syscall_64+0xc1/0x440 [ 2092.323269] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2092.325802] RIP: 0033:0x7f83e5f5c10b [ 2092.327631] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2092.337508] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2092.340377] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2092.343537] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2092.347404] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2092.350593] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2092.353702] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2096.057170] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 2096.067861] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2096.093663] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2096.098551] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2096.102565] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2096.106594] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2096.115972] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2096.119168] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2096.123138] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2096.130969] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2096.134890] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2096.138987] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2096.142591] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2096.146792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2096.150421] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2096.153961] Call Trace: [ 2096.155181] [ 2096.156605] ? show_regs.cold.9+0x22/0x2f [ 2096.158725] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2096.161246] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2096.163230] ? watchdog+0x40/0x40 [ 2096.171414] ? hrtimer_interrupt+0x128/0x2c0 [ 2096.173815] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2096.177707] ? apic_timer_interrupt+0xf/0x20 [ 2096.179886] [ 2096.180920] ? native_safe_halt+0x2e/0x50 [ 2096.182921] kvm_wait+0xcd/0xe0 [ 2096.184362] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2096.187480] do_raw_spin_lock+0x177/0x190 [ 2096.189473] _raw_spin_lock+0x21/0x30 [ 2096.191382] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2096.194169] obd_mod_free+0x46/0x140 [ptlrpc] [ 2096.196364] mdc_commit_open+0x91/0xb0 [mdc] [ 2096.198639] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2096.201608] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2096.204414] after_reply+0xe2c/0x1780 [ptlrpc] [ 2096.207353] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2096.209696] ? __next_timer_interrupt+0x160/0x160 [ 2096.211783] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2096.214373] ? do_wait_intr+0xf0/0xf0 [ 2096.215773] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2096.218774] kthread+0x1d1/0x200 [ 2096.220803] ? set_kthread_struct+0x70/0x70 [ 2096.222696] ret_from_fork+0x1f/0x30 [ 2120.066197] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2120.067661] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 2120.074831] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2120.074932] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2120.074942] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2120.075010] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2120.075028] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2120.075033] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2120.075038] RAX: 0000000000007d8e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2120.075040] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2120.075042] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2120.075045] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2120.075047] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2120.075050] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2120.075053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2120.075055] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2120.075063] Call Trace: [ 2120.075069] [ 2120.075076] ? show_regs.cold.9+0x22/0x2f [ 2120.075090] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2120.075098] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2120.076738] ? watchdog+0x40/0x40 [ 2120.076745] ? hrtimer_interrupt+0x128/0x2c0 [ 2120.076754] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2120.076764] ? apic_timer_interrupt+0xf/0x20 [ 2120.076773] [ 2120.076778] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2120.076785] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2120.076790] do_raw_spin_lock+0x177/0x190 [ 2120.076797] _raw_spin_lock+0x21/0x30 [ 2120.076806] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2120.078370] ? do_raw_spin_unlock+0x75/0x190 [ 2120.078380] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2120.078655] ? do_wait_intr+0xf0/0xf0 [ 2120.078793] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2120.081894] mdc_reint+0x2d/0x110 [mdc] [ 2120.081941] mdc_unlink+0x658/0x870 [mdc] [ 2120.081973] lmv_unlink+0x34d/0xbd0 [lmv] [ 2120.082019] ll_unlink+0x24d/0x7f0 [lustre] [ 2120.082162] vfs_unlink+0x202/0x330 [ 2120.082176] do_unlinkat+0x2fa/0x460 [ 2120.082181] __x64_sys_unlink+0x37/0x50 [ 2120.082189] do_syscall_64+0xc1/0x440 [ 2120.082196] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2120.082207] RIP: 0033:0x7f83e5f5c10b [ 2120.082217] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2120.082226] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2120.082232] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2120.082235] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2120.082237] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2120.082239] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2120.082241] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2120.353072] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2120.392974] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2120.411417] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2120.421415] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2120.424917] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2120.430562] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2120.446938] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2120.450952] RAX: 0000000000007b92 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2120.455574] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2120.467894] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2120.479519] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2120.491133] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2120.495867] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2120.504589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2120.508679] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2120.514997] Call Trace: [ 2120.517189] [ 2120.518452] ? show_regs.cold.9+0x22/0x2f [ 2120.520984] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2120.523830] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2120.526537] ? watchdog+0x40/0x40 [ 2120.528769] ? hrtimer_interrupt+0x128/0x2c0 [ 2120.535472] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2120.538011] ? apic_timer_interrupt+0xf/0x20 [ 2120.542275] [ 2120.545037] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2120.551182] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2120.554368] do_raw_spin_lock+0x177/0x190 [ 2120.555889] _raw_spin_lock+0x21/0x30 [ 2120.565601] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2120.567655] process_one_work+0x2c8/0x700 [ 2120.578630] worker_thread+0x296/0x6e0 [ 2120.580147] ? rescuer_thread+0x570/0x570 [ 2120.589114] kthread+0x1d1/0x200 [ 2120.590804] ? set_kthread_struct+0x70/0x70 [ 2120.595968] ret_from_fork+0x1f/0x30 [ 2124.056029] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 2124.062210] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2124.088393] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2124.093472] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2124.097927] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2124.100525] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2124.111238] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2124.116082] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2124.118979] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2124.123545] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2124.127950] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2124.131746] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2124.135565] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2124.138875] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2124.141783] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2124.145278] Call Trace: [ 2124.146744] [ 2124.147861] ? show_regs.cold.9+0x22/0x2f [ 2124.149881] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2124.152282] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2124.154540] ? watchdog+0x40/0x40 [ 2124.156253] ? hrtimer_interrupt+0x128/0x2c0 [ 2124.158586] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2124.161251] ? apic_timer_interrupt+0xf/0x20 [ 2124.163218] [ 2124.164029] ? native_safe_halt+0x2e/0x50 [ 2124.165946] kvm_wait+0xcd/0xe0 [ 2124.167777] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2124.170819] do_raw_spin_lock+0x177/0x190 [ 2124.172930] _raw_spin_lock+0x21/0x30 [ 2124.175036] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2124.177130] obd_mod_free+0x46/0x140 [ptlrpc] [ 2124.180032] mdc_commit_open+0x91/0xb0 [mdc] [ 2124.182230] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2124.185700] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2124.188738] after_reply+0xe2c/0x1780 [ptlrpc] [ 2124.191168] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2124.194063] ? __next_timer_interrupt+0x160/0x160 [ 2124.196501] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2124.199133] ? do_wait_intr+0xf0/0xf0 [ 2124.202071] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2124.205232] kthread+0x1d1/0x200 [ 2124.206978] ? set_kthread_struct+0x70/0x70 [ 2124.209134] ret_from_fork+0x1f/0x30 [ 2148.060244] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 2148.062802] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2148.064568] Modules linked in: [ 2148.070262] Modules linked in: [ 2148.070265] lustre(O) osp(O) [ 2148.072395] lustre(O) [ 2148.073739] ofd(O) [ 2148.074736] osp(O) [ 2148.075985] lod(O) [ 2148.076721] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) [ 2148.077830] mdt(O) [ 2148.078536] obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2148.078574] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2148.078580] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2148.078583] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2148.078598] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2148.078601] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2148.078604] RAX: 0000000000007f85 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2148.078605] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2148.078607] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2148.078608] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2148.078610] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2148.078613] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2148.078615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2148.078616] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2148.078623] Call Trace: [ 2148.078627] [ 2148.078630] ? show_regs.cold.9+0x22/0x2f [ 2148.078637] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2148.078642] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2148.078645] ? watchdog+0x40/0x40 [ 2148.078647] ? hrtimer_interrupt+0x128/0x2c0 [ 2148.078650] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2148.078656] ? apic_timer_interrupt+0xf/0x20 [ 2148.078659] [ 2148.078661] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2148.078664] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2148.078666] do_raw_spin_lock+0x177/0x190 [ 2148.078670] _raw_spin_lock+0x21/0x30 [ 2148.078675] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2148.086016] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2148.086095] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2148.086101] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2148.086107] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2148.086838] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2148.086850] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2148.086854] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2148.086858] RAX: 0000000000007f09 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2148.086860] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2148.086862] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2148.086864] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2148.086867] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2148.086870] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2148.086873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2148.086875] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2148.086889] Call Trace: [ 2148.086901] [ 2148.086906] ? show_regs.cold.9+0x22/0x2f [ 2148.086916] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2148.086923] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2148.086927] ? watchdog+0x40/0x40 [ 2148.086930] ? hrtimer_interrupt+0x128/0x2c0 [ 2148.086933] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2148.086941] ? apic_timer_interrupt+0xf/0x20 [ 2148.086945] [ 2148.086948] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2148.086952] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2148.086958] do_raw_spin_lock+0x177/0x190 [ 2148.086962] _raw_spin_lock+0x21/0x30 [ 2148.086967] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2148.087787] ? do_raw_spin_unlock+0x75/0x190 [ 2148.087797] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2148.100481] process_one_work+0x2c8/0x700 [ 2148.105471] ? do_wait_intr+0xf0/0xf0 [ 2148.109930] worker_thread+0x296/0x6e0 [ 2148.113614] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2148.128218] ? rescuer_thread+0x570/0x570 [ 2148.128238] kthread+0x1d1/0x200 [ 2148.128243] ? set_kthread_struct+0x70/0x70 [ 2148.128248] ret_from_fork+0x1f/0x30 [ 2148.465615] mdc_reint+0x2d/0x110 [mdc] [ 2148.470018] mdc_unlink+0x658/0x870 [mdc] [ 2148.476968] lmv_unlink+0x34d/0xbd0 [lmv] [ 2148.480590] ll_unlink+0x24d/0x7f0 [lustre] [ 2148.483304] vfs_unlink+0x202/0x330 [ 2148.488180] do_unlinkat+0x2fa/0x460 [ 2148.491822] __x64_sys_unlink+0x37/0x50 [ 2148.496754] do_syscall_64+0xc1/0x440 [ 2148.498044] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2148.499885] RIP: 0033:0x7f83e5f5c10b [ 2148.513876] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2148.534997] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2148.544196] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2148.547289] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2148.550507] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2148.556440] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2148.569039] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2152.056093] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 2152.060494] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2152.093660] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2152.101486] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2152.111273] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2152.123665] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2152.153620] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2152.167258] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2152.180619] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2152.192196] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2152.203536] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2152.207609] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2152.214351] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2152.224831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2152.231426] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2152.237392] Call Trace: [ 2152.240317] [ 2152.240913] ? show_regs.cold.9+0x22/0x2f [ 2152.246143] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2152.250698] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2152.257830] ? watchdog+0x40/0x40 [ 2152.258957] ? hrtimer_interrupt+0x128/0x2c0 [ 2152.267965] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2152.271262] ? apic_timer_interrupt+0xf/0x20 [ 2152.276142] [ 2152.278499] ? native_safe_halt+0x2e/0x50 [ 2152.282499] kvm_wait+0xcd/0xe0 [ 2152.283653] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2152.286546] do_raw_spin_lock+0x177/0x190 [ 2152.292222] _raw_spin_lock+0x21/0x30 [ 2152.294328] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2152.296981] obd_mod_free+0x46/0x140 [ptlrpc] [ 2152.298989] mdc_commit_open+0x91/0xb0 [mdc] [ 2152.301093] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2152.303417] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2152.305171] after_reply+0xe2c/0x1780 [ptlrpc] [ 2152.307107] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2152.309264] ? __next_timer_interrupt+0x160/0x160 [ 2152.311320] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2152.312780] ? do_wait_intr+0xf0/0xf0 [ 2152.313971] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2152.316528] kthread+0x1d1/0x200 [ 2152.317700] ? set_kthread_struct+0x70/0x70 [ 2152.319567] ret_from_fork+0x1f/0x30 [ 2176.060033] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 2176.062598] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2176.063774] Modules linked in: lustre(O) [ 2176.066557] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2176.066624] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2176.066630] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2176.066633] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2176.066645] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2176.066649] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2176.066653] RAX: 00000000000051d7 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2176.066655] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2176.066657] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2176.066659] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2176.066661] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2176.066663] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2176.066666] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2176.066668] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2176.066675] Call Trace: [ 2176.066678] [ 2176.066682] ? show_regs.cold.9+0x22/0x2f [ 2176.066690] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2176.066695] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2176.066699] ? watchdog+0x40/0x40 [ 2176.066702] ? hrtimer_interrupt+0x128/0x2c0 [ 2176.066705] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2176.066712] ? apic_timer_interrupt+0xf/0x20 [ 2176.066716] [ 2176.066770] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2176.066776] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2176.066781] do_raw_spin_lock+0x177/0x190 [ 2176.066786] _raw_spin_lock+0x21/0x30 [ 2176.066791] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2176.068894] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr [ 2176.096198] ? do_raw_spin_unlock+0x75/0x190 [ 2176.100213] i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2176.100244] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2176.100251] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2176.100257] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2176.110810] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2176.113422] [ 2176.113430] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2176.121460] ? do_wait_intr+0xf0/0xf0 [ 2176.124608] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2176.128725] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2176.131580] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2176.133996] mdc_reint+0x2d/0x110 [mdc] [ 2176.136981] RAX: 0000000000003fc2 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2176.136992] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2176.136994] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2176.136996] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2176.136998] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2176.137001] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2176.137005] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2176.137007] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2176.137015] Call Trace: [ 2176.137025] [ 2176.137029] ? show_regs.cold.9+0x22/0x2f [ 2176.137044] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2176.137051] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2176.137055] ? watchdog+0x40/0x40 [ 2176.137060] ? hrtimer_interrupt+0x128/0x2c0 [ 2176.137064] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2176.137078] ? apic_timer_interrupt+0xf/0x20 [ 2176.137083] [ 2176.137085] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2176.137091] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2176.137098] do_raw_spin_lock+0x177/0x190 [ 2176.137102] _raw_spin_lock+0x21/0x30 [ 2176.137108] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2176.140161] mdc_unlink+0x658/0x870 [mdc] [ 2176.143646] process_one_work+0x2c8/0x700 [ 2176.143668] worker_thread+0x296/0x6e0 [ 2176.143676] ? rescuer_thread+0x570/0x570 [ 2176.143680] kthread+0x1d1/0x200 [ 2176.145502] lmv_unlink+0x34d/0xbd0 [lmv] [ 2176.150607] ? set_kthread_struct+0x70/0x70 [ 2176.150622] ret_from_fork+0x1f/0x30 [ 2176.151649] ll_unlink+0x24d/0x7f0 [lustre] [ 2176.289514] vfs_unlink+0x202/0x330 [ 2176.290794] do_unlinkat+0x2fa/0x460 [ 2176.292425] __x64_sys_unlink+0x37/0x50 [ 2176.294195] do_syscall_64+0xc1/0x440 [ 2176.295617] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2176.297591] RIP: 0033:0x7f83e5f5c10b [ 2176.299042] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2176.307418] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2176.310535] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2176.313706] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2176.316748] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2176.319647] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2176.322869] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2180.056022] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 2180.060604] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2180.099437] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2180.110293] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2180.115630] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2180.121000] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2180.138658] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2180.144379] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2180.149238] RDX: 0000000000005745 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2180.154028] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2180.158357] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2180.163116] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2180.168303] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2180.174420] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2180.181490] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2180.187483] Call Trace: [ 2180.189503] [ 2180.191326] ? show_regs.cold.9+0x22/0x2f [ 2180.194212] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2180.198438] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2180.203386] ? watchdog+0x40/0x40 [ 2180.205639] ? hrtimer_interrupt+0x128/0x2c0 [ 2180.209307] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2180.212979] ? apic_timer_interrupt+0xf/0x20 [ 2180.217070] [ 2180.219315] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2180.224416] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2180.229475] do_raw_spin_lock+0x177/0x190 [ 2180.232545] _raw_spin_lock+0x21/0x30 [ 2180.235448] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2180.241901] obd_mod_free+0x46/0x140 [ptlrpc] [ 2180.246713] mdc_commit_open+0x91/0xb0 [mdc] [ 2180.251582] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2180.259513] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2180.266478] after_reply+0xe2c/0x1780 [ptlrpc] [ 2180.271348] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2180.282411] ? __next_timer_interrupt+0x160/0x160 [ 2180.286287] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2180.290281] ? do_wait_intr+0xf0/0xf0 [ 2180.292873] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2180.297141] kthread+0x1d1/0x200 [ 2180.298622] ? set_kthread_struct+0x70/0x70 [ 2180.302492] ret_from_fork+0x1f/0x30 [ 2204.062914] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 2204.065800] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2204.068235] Modules linked in: lustre(O) [ 2204.074982] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2204.075060] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2204.075067] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2204.075070] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2204.075086] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2204.075090] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2204.075094] RAX: 0000000000002e9e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2204.075096] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2204.075099] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2204.075100] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2204.075103] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2204.075106] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2204.075109] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2204.075111] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2204.075120] Call Trace: [ 2204.075125] [ 2204.075131] ? show_regs.cold.9+0x22/0x2f [ 2204.075141] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2204.075147] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2204.075151] ? watchdog+0x40/0x40 [ 2204.075156] ? hrtimer_interrupt+0x128/0x2c0 [ 2204.075159] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2204.075168] ? apic_timer_interrupt+0xf/0x20 [ 2204.075172] [ 2204.075175] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2204.075179] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2204.075183] do_raw_spin_lock+0x177/0x190 [ 2204.075187] _raw_spin_lock+0x21/0x30 [ 2204.075193] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2204.075475] ? do_raw_spin_unlock+0x75/0x190 [ 2204.075481] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2204.075688] ? do_wait_intr+0xf0/0xf0 [ 2204.075696] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2204.078659] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2204.078735] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2204.078743] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2204.078749] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2204.079021] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2204.079033] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2204.079038] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2204.079043] RAX: 00000000000035d5 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2204.079046] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2204.079049] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2204.079051] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2204.079053] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2204.079056] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2204.079059] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2204.079062] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2204.079071] Call Trace: [ 2204.079076] [ 2204.079082] ? show_regs.cold.9+0x22/0x2f [ 2204.118914] mdc_reint+0x2d/0x110 [mdc] [ 2204.118974] mdc_unlink+0x658/0x870 [mdc] [ 2204.119002] lmv_unlink+0x34d/0xbd0 [lmv] [ 2204.119036] ll_unlink+0x24d/0x7f0 [lustre] [ 2204.119163] vfs_unlink+0x202/0x330 [ 2204.119172] do_unlinkat+0x2fa/0x460 [ 2204.119178] __x64_sys_unlink+0x37/0x50 [ 2204.119182] do_syscall_64+0xc1/0x440 [ 2204.119189] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2204.119199] RIP: 0033:0x7f83e5f5c10b [ 2204.119206] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2204.119210] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2204.119216] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2204.119218] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2204.119221] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2204.119223] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2204.119225] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2204.383314] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2204.385021] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2204.387056] ? watchdog+0x40/0x40 [ 2204.388531] ? hrtimer_interrupt+0x128/0x2c0 [ 2204.390560] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2204.392522] ? apic_timer_interrupt+0xf/0x20 [ 2204.394377] [ 2204.395352] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2204.397631] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2204.400080] do_raw_spin_lock+0x177/0x190 [ 2204.401912] _raw_spin_lock+0x21/0x30 [ 2204.403717] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2204.406040] process_one_work+0x2c8/0x700 [ 2204.408038] worker_thread+0x296/0x6e0 [ 2204.409624] ? rescuer_thread+0x570/0x570 [ 2204.411619] kthread+0x1d1/0x200 [ 2204.413624] ? set_kthread_struct+0x70/0x70 [ 2204.415844] ret_from_fork+0x1f/0x30 [ 2208.061034] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 2208.069357] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2208.110598] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2208.127795] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2208.133325] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2208.140601] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2208.161378] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2208.170481] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2208.177218] RDX: 0000000000006a1f RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2208.185441] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2208.197025] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2208.206685] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2208.218586] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2208.221397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2208.237061] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2208.245241] Call Trace: [ 2208.251724] [ 2208.252401] ? show_regs.cold.9+0x22/0x2f [ 2208.257249] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2208.264385] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2208.269568] ? watchdog+0x40/0x40 [ 2208.276497] ? hrtimer_interrupt+0x128/0x2c0 [ 2208.281561] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2208.289493] ? apic_timer_interrupt+0xf/0x20 [ 2208.294206] [ 2208.299339] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2208.303909] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2208.309189] do_raw_spin_lock+0x177/0x190 [ 2208.316063] _raw_spin_lock+0x21/0x30 [ 2208.318055] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2208.324835] obd_mod_free+0x46/0x140 [ptlrpc] [ 2208.331238] mdc_commit_open+0x91/0xb0 [mdc] [ 2208.336046] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2208.346894] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2208.356511] after_reply+0xe2c/0x1780 [ptlrpc] [ 2208.361719] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2208.363836] ? __next_timer_interrupt+0x160/0x160 [ 2208.371918] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2208.380055] ? do_wait_intr+0xf0/0xf0 [ 2208.384870] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2208.391799] kthread+0x1d1/0x200 [ 2208.393238] ? set_kthread_struct+0x70/0x70 [ 2208.395156] ret_from_fork+0x1f/0x30 [ 2212.832080] INFO: task khugepaged:48 blocked for more than 120 seconds. [ 2212.835725] Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2212.839736] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2212.844483] task:khugepaged state:D stack:0 pid:48 ppid:2 flags:0x80004000 [ 2212.848729] Call Trace: [ 2212.850404] __schedule+0x351/0xcb0 [ 2212.852304] ? wait_for_completion+0xae/0x1e0 [ 2212.856521] schedule+0xc0/0x180 [ 2212.857842] schedule_timeout+0x126/0x190 [ 2212.859609] ? try_to_wake_up+0x80/0x710 [ 2212.861851] ? __prepare_to_swait+0x5b/0x90 [ 2212.864215] ? do_raw_spin_unlock+0x75/0x190 [ 2212.866820] wait_for_completion+0xf0/0x1e0 [ 2212.869184] flush_work+0x1f3/0x390 [ 2212.871215] ? flush_workqueue_prep_pwqs+0x200/0x200 [ 2212.873933] ? do_raw_spin_unlock+0x75/0x190 [ 2212.876803] ? __pagevec_release+0x50/0x50 [ 2212.879118] lru_add_drain_all+0x1d4/0x280 [ 2212.881321] khugepaged+0xa2/0x16f0 [ 2212.883230] ? woken_wake_function+0x30/0x30 [ 2212.885821] ? collapse_pte_mapped_thp+0x8a0/0x8a0 [ 2212.888819] kthread+0x1d1/0x200 [ 2212.890835] ? set_kthread_struct+0x70/0x70 [ 2212.893528] ret_from_fork+0x1f/0x30 [ 2212.895649] INFO: task writemany:36353 blocked for more than 120 seconds. [ 2212.899854] Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2212.903872] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2212.906651] task:writemany state:D stack:0 pid:36353 ppid:36350 flags:0x80000004 [ 2212.911047] Call Trace: [ 2212.912748] __schedule+0x351/0xcb0 [ 2212.915088] schedule+0xc0/0x180 [ 2212.916966] schedule_preempt_disabled+0x21/0x40 [ 2212.920058] rwsem_down_write_slowpath+0x5d7/0xa40 [ 2212.923570] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2212.927625] down_write+0x80/0xd0 [ 2212.929467] do_last+0x2eb/0xfc0 [ 2212.932576] ? nd_jump_root+0xe5/0x160 [ 2212.934754] ? path_init+0x437/0x520 [ 2212.936656] path_openat+0xf7/0x500 [ 2212.938602] ? do_raw_spin_unlock+0x75/0x190 [ 2212.940869] do_filp_open+0x99/0x140 [ 2212.942505] ? getname_flags+0x6e/0x330 [ 2212.944763] ? __check_object_size+0xff/0x256 [ 2212.947317] ? do_raw_spin_unlock+0x75/0x190 [ 2212.949923] ? _raw_spin_unlock+0x12/0x30 [ 2212.952678] do_sys_openat2+0x2b4/0x410 [ 2212.954979] do_sys_open+0x73/0xa0 [ 2212.956459] __x64_sys_openat+0x24/0x30 [ 2212.958330] do_syscall_64+0xc1/0x440 [ 2212.960448] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2212.963321] RIP: 0033:0x7f83e6051332 [ 2212.965323] Code: Unable to access opcode bytes at RIP 0x7f83e6051308. [ 2212.968829] RSP: 002b:00007fff7d2bcf90 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2212.972346] RAX: ffffffffffffffda RBX: 0000000000001538 RCX: 00007f83e6051332 [ 2212.976189] RDX: 0000000000000042 RSI: 00007fff7d2bd060 RDI: 00000000ffffff9c [ 2212.980018] RBP: 00007fff7d2bd460 R08: 0000000000000000 R09: 0000000000000000 [ 2212.984971] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff7d2bd060 [ 2212.989878] R13: 00000000ffffffff R14: 0000000000000011 R15: 0000000000000000 [ 2212.995835] INFO: task writemany:36354 blocked for more than 120 seconds. [ 2212.998457] Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2213.001725] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2213.004850] task:writemany state:D stack:0 pid:36354 ppid:36350 flags:0x80000004 [ 2213.008201] Call Trace: [ 2213.009324] __schedule+0x351/0xcb0 [ 2213.010915] schedule+0xc0/0x180 [ 2213.012238] schedule_preempt_disabled+0x21/0x40 [ 2213.014555] rwsem_down_write_slowpath+0x5d7/0xa40 [ 2213.016782] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2213.019459] down_write+0x80/0xd0 [ 2213.021127] do_last+0x2eb/0xfc0 [ 2213.022517] ? nd_jump_root+0xe5/0x160 [ 2213.024333] ? path_init+0x437/0x520 [ 2213.026032] path_openat+0xf7/0x500 [ 2213.027727] ? do_raw_spin_unlock+0x75/0x190 [ 2213.029524] do_filp_open+0x99/0x140 [ 2213.031144] ? getname_flags+0x6e/0x330 [ 2213.032824] ? __check_object_size+0xff/0x256 [ 2213.034606] ? do_raw_spin_unlock+0x75/0x190 [ 2213.035928] ? _raw_spin_unlock+0x12/0x30 [ 2213.037518] do_sys_openat2+0x2b4/0x410 [ 2213.038866] do_sys_open+0x73/0xa0 [ 2213.040317] __x64_sys_openat+0x24/0x30 [ 2213.042196] do_syscall_64+0xc1/0x440 [ 2213.043910] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2213.046177] RIP: 0033:0x7f83e6051332 [ 2213.047916] Code: Unable to access opcode bytes at RIP 0x7f83e6051308. [ 2213.050933] RSP: 002b:00007fff7d2bcf90 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2213.054329] RAX: ffffffffffffffda RBX: 000000000000a058 RCX: 00007f83e6051332 [ 2213.057306] RDX: 0000000000000042 RSI: 00007fff7d2bd060 RDI: 00000000ffffff9c [ 2213.060280] RBP: 00007fff7d2bd460 R08: 0000000000000000 R09: 0000000000000000 [ 2213.063177] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff7d2bd060 [ 2213.066482] R13: 00000000ffffffff R14: 000000000000000f R15: 0000000000000000 [ 2213.069929] INFO: task writemany:36355 blocked for more than 120 seconds. [ 2213.073499] Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2213.076978] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2213.080341] task:writemany state:D stack:0 pid:36355 ppid:36350 flags:0x80000004 [ 2213.084043] Call Trace: [ 2213.085288] __schedule+0x351/0xcb0 [ 2213.089900] schedule+0xc0/0x180 [ 2213.091766] schedule_preempt_disabled+0x21/0x40 [ 2213.097547] rwsem_down_write_slowpath+0x5d7/0xa40 [ 2213.101266] ? __might_sleep+0x59/0xc0 [ 2213.103297] down_write+0x80/0xd0 [ 2213.104986] do_unlinkat+0x184/0x460 [ 2213.107837] __x64_sys_unlink+0x37/0x50 [ 2213.110262] do_syscall_64+0xc1/0x440 [ 2213.112524] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2213.115468] RIP: 0033:0x7f83e5f5c10b [ 2213.117528] Code: Unable to access opcode bytes at RIP 0x7f83e5f5c0e1. [ 2213.120628] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2213.124269] RAX: ffffffffffffffda RBX: 000000000002c258 RCX: 00007f83e5f5c10b [ 2213.127833] RDX: 0000000000000038 RSI: 00007fff7d2bd460 RDI: 00007fff7d2bd060 [ 2213.131277] RBP: 00007fff7d2bd460 R08: 00007f83e62f421c R09: 00007f83e62f4260 [ 2213.134580] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2213.138156] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2213.145131] INFO: task writemany:36356 blocked for more than 120 seconds. [ 2213.148437] Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2213.151856] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2213.155682] task:writemany state:D stack:0 pid:36356 ppid:36350 flags:0x80000004 [ 2213.158921] Call Trace: [ 2213.159812] __schedule+0x351/0xcb0 [ 2213.161499] schedule+0xc0/0x180 [ 2213.162735] schedule_preempt_disabled+0x21/0x40 [ 2213.164516] rwsem_down_write_slowpath+0x5d7/0xa40 [ 2213.166450] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2213.168850] down_write+0x80/0xd0 [ 2213.170166] do_last+0x2eb/0xfc0 [ 2213.171533] ? nd_jump_root+0xe5/0x160 [ 2213.173200] ? path_init+0x437/0x520 [ 2213.174862] path_openat+0xf7/0x500 [ 2213.176186] ? do_raw_spin_unlock+0x75/0x190 [ 2213.178154] do_filp_open+0x99/0x140 [ 2213.179816] ? getname_flags+0x6e/0x330 [ 2213.181498] ? __check_object_size+0xff/0x256 [ 2213.183676] ? do_raw_spin_unlock+0x75/0x190 [ 2213.185718] ? _raw_spin_unlock+0x12/0x30 [ 2213.187050] do_sys_openat2+0x2b4/0x410 [ 2213.188548] do_sys_open+0x73/0xa0 [ 2213.190018] __x64_sys_openat+0x24/0x30 [ 2213.191596] do_syscall_64+0xc1/0x440 [ 2213.193283] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2213.195615] RIP: 0033:0x7f83e6051332 [ 2213.197360] Code: Unable to access opcode bytes at RIP 0x7f83e6051308. [ 2213.200338] RSP: 002b:00007fff7d2bcf90 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2213.203584] RAX: ffffffffffffffda RBX: 000000000008ebc0 RCX: 00007f83e6051332 [ 2213.206044] RDX: 0000000000000042 RSI: 00007fff7d2bd060 RDI: 00000000ffffff9c [ 2213.209621] RBP: 00007fff7d2bd460 R08: 0000000000000000 R09: 0000000000000000 [ 2213.212977] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007fff7d2bd060 [ 2213.216341] R13: 00000000ffffffff R14: 0000000000000002 R15: 0000000000000000 [ 2220.964088] rcu: INFO: rcu_sched self-detected stall on CPU [ 2220.971133] rcu: 0-....: (154838 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=56612 [ 2220.986944] (t=240027 jiffies g=262441 q=55817) [ 2220.991811] NMI backtrace for cpu 0 [ 2220.994526] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2221.004761] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2221.012491] Call Trace: [ 2221.013948] [ 2221.015421] ? dump_stack+0xbb/0x10e [ 2221.017618] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2221.021865] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2221.024124] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2221.027570] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2221.030489] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2221.032197] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2221.033968] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2221.040257] ? update_process_times+0x6d/0xc0 [ 2221.043562] ? tick_sched_handle.isra.18+0x27/0x90 [ 2221.046274] ? tick_sched_timer+0x53/0xb0 [ 2221.048502] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2221.051225] ? tick_sched_handle.isra.18+0x90/0x90 [ 2221.055221] ? hrtimer_interrupt+0x128/0x2c0 [ 2221.059154] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2221.062522] ? apic_timer_interrupt+0xf/0x20 [ 2221.065460] [ 2221.066896] ? native_safe_halt+0x2e/0x50 [ 2221.069522] ? kvm_wait+0xcd/0xe0 [ 2221.070795] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2221.072722] ? do_raw_spin_lock+0x177/0x190 [ 2221.078800] ? _raw_spin_lock+0x21/0x30 [ 2221.082508] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2221.086491] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2221.093496] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2221.097965] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2221.100275] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2221.102991] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2221.106104] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2221.108266] ? __next_timer_interrupt+0x160/0x160 [ 2221.111144] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2221.113481] ? do_wait_intr+0xf0/0xf0 [ 2221.115251] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2221.120342] ? kthread+0x1d1/0x200 [ 2221.122108] ? set_kthread_struct+0x70/0x70 [ 2221.124771] ? ret_from_fork+0x1f/0x30 [ 2232.063009] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 2232.063114] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2232.069757] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2232.069847] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2232.069855] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2232.069863] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2232.070179] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2232.070191] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2232.070196] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2232.070202] RAX: 00000000000072b1 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2232.070204] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2232.070208] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2232.070210] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2232.070213] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2232.070217] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2232.070220] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2232.070223] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2232.070231] Call Trace: [ 2232.070237] [ 2232.070242] ? show_regs.cold.9+0x22/0x2f [ 2232.070252] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2232.070259] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2232.070263] ? watchdog+0x40/0x40 [ 2232.070267] ? hrtimer_interrupt+0x128/0x2c0 [ 2232.070271] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2232.070281] ? apic_timer_interrupt+0xf/0x20 [ 2232.070286] [ 2232.070289] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2232.070293] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2232.070298] do_raw_spin_lock+0x177/0x190 [ 2232.070931] _raw_spin_lock+0x21/0x30 [ 2232.070937] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2232.071228] process_one_work+0x2c8/0x700 [ 2232.071235] worker_thread+0x296/0x6e0 [ 2232.071240] ? rescuer_thread+0x570/0x570 [ 2232.071244] kthread+0x1d1/0x200 [ 2232.071259] ? set_kthread_struct+0x70/0x70 [ 2232.071262] ret_from_fork+0x1f/0x30 [ 2232.319315] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2232.369801] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2232.382437] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2232.392859] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2232.398569] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2232.423514] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2232.431980] RAX: 0000000000007d1c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2232.439930] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2232.447762] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2232.452730] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2232.458737] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2232.465022] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2232.478964] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2232.483663] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2232.491076] Call Trace: [ 2232.492263] [ 2232.495996] ? show_regs.cold.9+0x22/0x2f [ 2232.499869] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2232.505901] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2232.509694] ? watchdog+0x40/0x40 [ 2232.512354] ? hrtimer_interrupt+0x128/0x2c0 [ 2232.515823] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2232.520381] ? apic_timer_interrupt+0xf/0x20 [ 2232.523678] [ 2232.525694] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2232.530165] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2232.532990] do_raw_spin_lock+0x177/0x190 [ 2232.537208] _raw_spin_lock+0x21/0x30 [ 2232.540172] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2232.544544] ? do_raw_spin_unlock+0x75/0x190 [ 2232.546743] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2232.550874] ? do_wait_intr+0xf0/0xf0 [ 2232.553156] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2232.558311] mdc_reint+0x2d/0x110 [mdc] [ 2232.561726] mdc_unlink+0x658/0x870 [mdc] [ 2232.564126] lmv_unlink+0x34d/0xbd0 [lmv] [ 2232.565697] ll_unlink+0x24d/0x7f0 [lustre] [ 2232.572986] vfs_unlink+0x202/0x330 [ 2232.576201] do_unlinkat+0x2fa/0x460 [ 2232.580191] __x64_sys_unlink+0x37/0x50 [ 2232.586541] do_syscall_64+0xc1/0x440 [ 2232.591369] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2232.599139] RIP: 0033:0x7f83e5f5c10b [ 2232.606876] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2232.622741] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2232.638262] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2232.644520] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2232.654957] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2232.662356] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2232.668490] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2248.056226] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 2248.060597] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2248.092571] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2248.106897] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2248.115746] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2248.123824] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2248.137288] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2248.142235] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2248.146773] RDX: 00000000000008a7 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2248.150767] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2248.155526] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2248.159473] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2248.163760] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2248.168565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2248.171799] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2248.176263] Call Trace: [ 2248.177624] [ 2248.178738] ? show_regs.cold.9+0x22/0x2f [ 2248.181533] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2248.184589] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2248.187287] ? watchdog+0x40/0x40 [ 2248.189262] ? hrtimer_interrupt+0x128/0x2c0 [ 2248.191728] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2248.194490] ? apic_timer_interrupt+0xf/0x20 [ 2248.196013] [ 2248.197668] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2248.201773] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2248.205494] do_raw_spin_lock+0x177/0x190 [ 2248.208672] _raw_spin_lock+0x21/0x30 [ 2248.210127] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2248.214193] obd_mod_free+0x46/0x140 [ptlrpc] [ 2248.224862] mdc_commit_open+0x91/0xb0 [mdc] [ 2248.227622] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2248.234309] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2248.241351] after_reply+0xe2c/0x1780 [ptlrpc] [ 2248.246782] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2248.251683] ? __next_timer_interrupt+0x160/0x160 [ 2248.255599] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2248.260196] ? do_wait_intr+0xf0/0xf0 [ 2248.263182] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2248.267541] kthread+0x1d1/0x200 [ 2248.269638] ? set_kthread_struct+0x70/0x70 [ 2248.271862] ret_from_fork+0x1f/0x30 [ 2260.060038] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 2260.062070] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2260.063520] Modules linked in: lustre(O) [ 2260.069693] Modules linked in: [ 2260.069694] osp(O) ofd(O) lod(O) mdt(O) [ 2260.070919] lustre(O) [ 2260.074924] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2260.074998] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2260.075005] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2260.075011] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2260.077934] osp(O) [ 2260.078767] [ 2260.078770] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2260.102500] ofd(O) [ 2260.107867] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2260.111995] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2260.114638] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2260.114650] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2260.114652] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2260.114654] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2260.114656] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2260.114658] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2260.114661] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2260.114664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2260.114666] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2260.114674] Call Trace: [ 2260.114680] [ 2260.114687] ? show_regs.cold.9+0x22/0x2f [ 2260.115286] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2260.115292] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2260.115295] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2260.115305] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2260.115308] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2260.115312] RAX: 0000000000004919 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2260.115314] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2260.115316] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2260.115317] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2260.115319] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2260.115321] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2260.115324] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2260.115325] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2260.115331] Call Trace: [ 2260.115334] [ 2260.115337] ? show_regs.cold.9+0x22/0x2f [ 2260.115344] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2260.115350] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2260.115353] ? watchdog+0x40/0x40 [ 2260.115357] ? hrtimer_interrupt+0x128/0x2c0 [ 2260.115360] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2260.115368] ? apic_timer_interrupt+0xf/0x20 [ 2260.115372] [ 2260.115375] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2260.115379] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2260.115383] do_raw_spin_lock+0x177/0x190 [ 2260.118501] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2260.120703] _raw_spin_lock+0x21/0x30 [ 2260.129911] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2260.129935] ? watchdog+0x40/0x40 [ 2260.129940] ? hrtimer_interrupt+0x128/0x2c0 [ 2260.129945] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2260.129953] ? apic_timer_interrupt+0xf/0x20 [ 2260.129958] [ 2260.129962] ? native_safe_halt+0x2e/0x50 [ 2260.129966] kvm_wait+0xcd/0xe0 [ 2260.129974] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2260.129984] do_raw_spin_lock+0x177/0x190 [ 2260.129989] _raw_spin_lock+0x21/0x30 [ 2260.129993] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2260.130396] process_one_work+0x2c8/0x700 [ 2260.130409] worker_thread+0x296/0x6e0 [ 2260.130413] ? rescuer_thread+0x570/0x570 [ 2260.130417] kthread+0x1d1/0x200 [ 2260.130421] ? set_kthread_struct+0x70/0x70 [ 2260.130424] ret_from_fork+0x1f/0x30 [ 2260.380936] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2260.384839] ? do_raw_spin_unlock+0x75/0x190 [ 2260.387491] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2260.391609] ? do_wait_intr+0xf0/0xf0 [ 2260.393866] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2260.397915] mdc_reint+0x2d/0x110 [mdc] [ 2260.398908] mdc_unlink+0x658/0x870 [mdc] [ 2260.401979] lmv_unlink+0x34d/0xbd0 [lmv] [ 2260.405784] ll_unlink+0x24d/0x7f0 [lustre] [ 2260.411090] vfs_unlink+0x202/0x330 [ 2260.412110] do_unlinkat+0x2fa/0x460 [ 2260.413704] __x64_sys_unlink+0x37/0x50 [ 2260.417521] do_syscall_64+0xc1/0x440 [ 2260.418935] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2260.421362] RIP: 0033:0x7f83e5f5c10b [ 2260.423744] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2260.440676] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2260.448870] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2260.451939] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2260.456845] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2260.466489] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2260.468534] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2276.056080] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 2276.059504] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2276.082254] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2276.087673] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2276.092410] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2276.094969] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2276.102562] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2276.106370] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2276.109657] RDX: 0000000000001db2 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2276.112262] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2276.115031] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2276.118037] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2276.121034] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2276.124489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2276.126924] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2276.129881] Call Trace: [ 2276.130639] [ 2276.131441] ? show_regs.cold.9+0x22/0x2f [ 2276.132697] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2276.134346] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2276.135903] ? watchdog+0x40/0x40 [ 2276.137245] ? hrtimer_interrupt+0x128/0x2c0 [ 2276.138917] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2276.144121] ? apic_timer_interrupt+0xf/0x20 [ 2276.146057] [ 2276.147303] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2276.150132] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2276.152936] do_raw_spin_lock+0x177/0x190 [ 2276.154893] _raw_spin_lock+0x21/0x30 [ 2276.156477] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2276.159067] obd_mod_free+0x46/0x140 [ptlrpc] [ 2276.161538] mdc_commit_open+0x91/0xb0 [mdc] [ 2276.163844] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2276.166639] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2276.169688] after_reply+0xe2c/0x1780 [ptlrpc] [ 2276.171818] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2276.174847] ? __next_timer_interrupt+0x160/0x160 [ 2276.177263] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2276.181451] ? do_wait_intr+0xf0/0xf0 [ 2276.186448] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2276.189741] kthread+0x1d1/0x200 [ 2276.193349] ? set_kthread_struct+0x70/0x70 [ 2276.195129] ret_from_fork+0x1f/0x30 [ 2288.060171] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 2288.063374] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2288.064055] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2288.081495] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2288.081508] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2288.081514] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2288.081878] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2288.081890] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2288.081894] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2288.081898] RAX: 00000000000010c0 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2288.081900] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2288.081902] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2288.081904] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2288.081907] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2288.081910] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2288.081913] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2288.081915] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2288.081924] Call Trace: [ 2288.081930] [ 2288.081936] ? show_regs.cold.9+0x22/0x2f [ 2288.081946] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2288.081952] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2288.081955] ? watchdog+0x40/0x40 [ 2288.081969] ? hrtimer_interrupt+0x128/0x2c0 [ 2288.081973] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2288.081982] ? apic_timer_interrupt+0xf/0x20 [ 2288.081986] [ 2288.081988] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2288.081993] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2288.081998] do_raw_spin_lock+0x177/0x190 [ 2288.082002] _raw_spin_lock+0x21/0x30 [ 2288.082006] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2288.082217] process_one_work+0x2c8/0x700 [ 2288.082223] worker_thread+0x296/0x6e0 [ 2288.082227] ? rescuer_thread+0x570/0x570 [ 2288.082231] kthread+0x1d1/0x200 [ 2288.082234] ? set_kthread_struct+0x70/0x70 [ 2288.082238] ret_from_fork+0x1f/0x30 [ 2288.222630] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2288.250100] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2288.260989] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2288.265935] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2288.267570] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2288.281904] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2288.284659] RAX: 0000000000000f8b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2288.292606] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2288.299118] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2288.301713] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2288.303618] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2288.307737] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2288.310847] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2288.312942] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2288.323585] Call Trace: [ 2288.324613] [ 2288.325567] ? show_regs.cold.9+0x22/0x2f [ 2288.328904] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2288.330605] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2288.332598] ? watchdog+0x40/0x40 [ 2288.334515] ? hrtimer_interrupt+0x128/0x2c0 [ 2288.336091] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2288.337799] ? apic_timer_interrupt+0xf/0x20 [ 2288.339430] [ 2288.340446] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2288.342597] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2288.344772] do_raw_spin_lock+0x177/0x190 [ 2288.346191] _raw_spin_lock+0x21/0x30 [ 2288.347868] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2288.350397] ? do_raw_spin_unlock+0x75/0x190 [ 2288.352368] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2288.354587] ? do_wait_intr+0xf0/0xf0 [ 2288.356303] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2288.359098] mdc_reint+0x2d/0x110 [mdc] [ 2288.360841] mdc_unlink+0x658/0x870 [mdc] [ 2288.362685] lmv_unlink+0x34d/0xbd0 [lmv] [ 2288.364846] ll_unlink+0x24d/0x7f0 [lustre] [ 2288.367529] vfs_unlink+0x202/0x330 [ 2288.368701] do_unlinkat+0x2fa/0x460 [ 2288.370431] __x64_sys_unlink+0x37/0x50 [ 2288.373067] do_syscall_64+0xc1/0x440 [ 2288.374613] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2288.377380] RIP: 0033:0x7f83e5f5c10b [ 2288.378577] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2288.387390] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2288.390712] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2288.394696] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2288.397773] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2288.401502] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2288.404511] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2304.057072] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 2304.063221] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2304.116360] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2304.134949] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2304.152665] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2304.158783] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2304.171132] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2304.176342] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2304.183595] RDX: 0000000000000572 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2304.187152] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2304.194115] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2304.197101] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2304.199984] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2304.215781] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2304.217849] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2304.224148] Call Trace: [ 2304.228410] [ 2304.230236] ? show_regs.cold.9+0x22/0x2f [ 2304.231666] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2304.234264] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2304.239410] ? watchdog+0x40/0x40 [ 2304.242867] ? hrtimer_interrupt+0x128/0x2c0 [ 2304.244539] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2304.249293] ? apic_timer_interrupt+0xf/0x20 [ 2304.253199] [ 2304.255940] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2304.261384] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2304.263184] do_raw_spin_lock+0x177/0x190 [ 2304.265954] _raw_spin_lock+0x21/0x30 [ 2304.267305] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2304.272715] obd_mod_free+0x46/0x140 [ptlrpc] [ 2304.274719] mdc_commit_open+0x91/0xb0 [mdc] [ 2304.277096] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2304.278943] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2304.291440] after_reply+0xe2c/0x1780 [ptlrpc] [ 2304.298445] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2304.301969] ? __next_timer_interrupt+0x160/0x160 [ 2304.307686] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2304.309574] ? do_wait_intr+0xf0/0xf0 [ 2304.318455] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2304.320385] kthread+0x1d1/0x200 [ 2304.323311] ? set_kthread_struct+0x70/0x70 [ 2304.326477] ret_from_fork+0x1f/0x30 [ 2316.060614] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 2316.062141] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 2316.063573] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2316.063662] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2316.063668] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2316.063674] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2316.067794] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2316.067872] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2316.067879] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2316.067881] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2316.067894] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2316.067898] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2316.067902] RAX: 0000000000007f86 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2316.067904] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2316.067907] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2316.067909] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2316.067911] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2316.067914] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2316.067917] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2316.067919] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2316.067927] Call Trace: [ 2316.067931] [ 2316.067936] ? show_regs.cold.9+0x22/0x2f [ 2316.067944] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2316.067950] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2316.067954] ? watchdog+0x40/0x40 [ 2316.067958] ? hrtimer_interrupt+0x128/0x2c0 [ 2316.067962] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2316.067970] ? apic_timer_interrupt+0xf/0x20 [ 2316.067974] [ 2316.067977] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2316.067981] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2316.067985] do_raw_spin_lock+0x177/0x190 [ 2316.067989] _raw_spin_lock+0x21/0x30 [ 2316.067994] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2316.068269] ? do_raw_spin_unlock+0x75/0x190 [ 2316.068275] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2316.068614] ? do_wait_intr+0xf0/0xf0 [ 2316.068622] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2316.068848] mdc_reint+0x2d/0x110 [mdc] [ 2316.068888] mdc_unlink+0x658/0x870 [mdc] [ 2316.068914] lmv_unlink+0x34d/0xbd0 [lmv] [ 2316.068949] ll_unlink+0x24d/0x7f0 [lustre] [ 2316.069087] vfs_unlink+0x202/0x330 [ 2316.069097] do_unlinkat+0x2fa/0x460 [ 2316.069101] __x64_sys_unlink+0x37/0x50 [ 2316.069105] do_syscall_64+0xc1/0x440 [ 2316.069113] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2316.069119] RIP: 0033:0x7f83e5f5c10b [ 2316.069125] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2316.069129] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2316.069133] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2316.069135] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2316.069138] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2316.069144] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2316.069146] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2316.361473] [ 2316.364196] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2316.366048] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2316.373616] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2316.377352] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2316.381098] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2316.384330] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2316.387943] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2316.391026] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2316.396998] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2316.402622] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2316.412558] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2316.417839] Call Trace: [ 2316.422159] [ 2316.423674] ? show_regs.cold.9+0x22/0x2f [ 2316.425232] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2316.427115] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2316.431799] ? watchdog+0x40/0x40 [ 2316.434422] ? hrtimer_interrupt+0x128/0x2c0 [ 2316.437533] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2316.439959] ? apic_timer_interrupt+0xf/0x20 [ 2316.446229] [ 2316.449071] ? native_safe_halt+0x2e/0x50 [ 2316.451366] kvm_wait+0xcd/0xe0 [ 2316.453363] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2316.459230] do_raw_spin_lock+0x177/0x190 [ 2316.461437] _raw_spin_lock+0x21/0x30 [ 2316.465224] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2316.471344] process_one_work+0x2c8/0x700 [ 2316.475176] worker_thread+0x296/0x6e0 [ 2316.478464] ? rescuer_thread+0x570/0x570 [ 2316.480269] kthread+0x1d1/0x200 [ 2316.484943] ? set_kthread_struct+0x70/0x70 [ 2316.491260] ret_from_fork+0x1f/0x30 [ 2332.057814] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 2332.066232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2332.099760] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2332.108853] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2332.115145] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2332.121475] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2332.134465] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2332.139904] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2332.145327] RDX: 00000000000001fd RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2332.148188] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2332.150406] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2332.153407] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2332.156036] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2332.163968] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2332.166842] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2332.169905] Call Trace: [ 2332.170784] [ 2332.171496] ? show_regs.cold.9+0x22/0x2f [ 2332.173073] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2332.175201] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2332.177085] ? watchdog+0x40/0x40 [ 2332.178209] ? hrtimer_interrupt+0x128/0x2c0 [ 2332.181029] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2332.182930] ? apic_timer_interrupt+0xf/0x20 [ 2332.184779] [ 2332.193447] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2332.199494] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2332.201946] do_raw_spin_lock+0x177/0x190 [ 2332.203739] _raw_spin_lock+0x21/0x30 [ 2332.205848] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2332.208242] obd_mod_free+0x46/0x140 [ptlrpc] [ 2332.210197] mdc_commit_open+0x91/0xb0 [mdc] [ 2332.212223] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2332.219969] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2332.222925] after_reply+0xe2c/0x1780 [ptlrpc] [ 2332.225007] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2332.229948] ? __next_timer_interrupt+0x160/0x160 [ 2332.232064] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2332.234635] ? do_wait_intr+0xf0/0xf0 [ 2332.239342] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2332.241648] kthread+0x1d1/0x200 [ 2332.245211] ? set_kthread_struct+0x70/0x70 [ 2332.248594] ret_from_fork+0x1f/0x30 [ 2335.712098] INFO: task khugepaged:48 blocked for more than 120 seconds. [ 2335.719644] Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2335.732097] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2335.749825] task:khugepaged state:D stack:0 pid:48 ppid:2 flags:0x80004000 [ 2335.764182] Call Trace: [ 2335.765049] __schedule+0x351/0xcb0 [ 2335.766284] ? wait_for_completion+0xae/0x1e0 [ 2335.782960] schedule+0xc0/0x180 [ 2335.789674] schedule_timeout+0x126/0x190 [ 2335.793819] ? try_to_wake_up+0x80/0x710 [ 2335.803951] ? __prepare_to_swait+0x5b/0x90 [ 2335.812484] ? do_raw_spin_unlock+0x75/0x190 [ 2335.815399] wait_for_completion+0xf0/0x1e0 [ 2335.820941] flush_work+0x1f3/0x390 [ 2335.825358] ? flush_workqueue_prep_pwqs+0x200/0x200 [ 2335.830768] ? do_raw_spin_unlock+0x75/0x190 [ 2335.834041] ? __pagevec_release+0x50/0x50 [ 2335.838663] lru_add_drain_all+0x1d4/0x280 [ 2335.845393] khugepaged+0xa2/0x16f0 [ 2335.850907] ? woken_wake_function+0x30/0x30 [ 2335.857548] ? collapse_pte_mapped_thp+0x8a0/0x8a0 [ 2335.860436] kthread+0x1d1/0x200 [ 2335.861848] ? set_kthread_struct+0x70/0x70 [ 2335.864590] ret_from_fork+0x1f/0x30 [ 2344.060046] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 2344.063749] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 2344.075894] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2344.075977] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2344.075983] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2344.075989] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2344.076258] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2344.076269] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2344.076273] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2344.076278] RAX: 0000000000005f11 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2344.076281] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2344.076283] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2344.076285] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2344.076288] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2344.076290] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2344.076293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2344.076295] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2344.076304] Call Trace: [ 2344.076308] [ 2344.076313] ? show_regs.cold.9+0x22/0x2f [ 2344.076321] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2344.076328] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2344.076332] ? watchdog+0x40/0x40 [ 2344.076337] ? hrtimer_interrupt+0x128/0x2c0 [ 2344.076341] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2344.076349] ? apic_timer_interrupt+0xf/0x20 [ 2344.076354] [ 2344.076356] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2344.076361] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2344.076366] do_raw_spin_lock+0x177/0x190 [ 2344.076371] _raw_spin_lock+0x21/0x30 [ 2344.076376] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2344.076583] process_one_work+0x2c8/0x700 [ 2344.076590] worker_thread+0x296/0x6e0 [ 2344.076594] ? rescuer_thread+0x570/0x570 [ 2344.076598] kthread+0x1d1/0x200 [ 2344.076602] ? set_kthread_struct+0x70/0x70 [ 2344.076605] ret_from_fork+0x1f/0x30 [ 2344.467926] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2344.531144] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2344.539344] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2344.551766] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2344.559400] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2344.576935] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2344.582139] RAX: 0000000000007c19 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2344.590600] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2344.599301] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2344.610587] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2344.615347] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2344.623676] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2344.629887] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2344.635992] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2344.644145] Call Trace: [ 2344.648264] [ 2344.651367] ? show_regs.cold.9+0x22/0x2f [ 2344.658098] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2344.662942] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2344.671864] ? watchdog+0x40/0x40 [ 2344.673959] ? hrtimer_interrupt+0x128/0x2c0 [ 2344.678917] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2344.680415] ? apic_timer_interrupt+0xf/0x20 [ 2344.684353] [ 2344.690257] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2344.696636] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2344.703682] do_raw_spin_lock+0x177/0x190 [ 2344.712786] _raw_spin_lock+0x21/0x30 [ 2344.721739] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2344.734571] ? do_raw_spin_unlock+0x75/0x190 [ 2344.740633] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2344.749835] ? do_wait_intr+0xf0/0xf0 [ 2344.751119] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2344.766358] mdc_reint+0x2d/0x110 [mdc] [ 2344.769911] mdc_unlink+0x658/0x870 [mdc] [ 2344.773309] lmv_unlink+0x34d/0xbd0 [lmv] [ 2344.778846] ll_unlink+0x24d/0x7f0 [lustre] [ 2344.781032] vfs_unlink+0x202/0x330 [ 2344.783902] do_unlinkat+0x2fa/0x460 [ 2344.786827] __x64_sys_unlink+0x37/0x50 [ 2344.788994] do_syscall_64+0xc1/0x440 [ 2344.792129] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2344.799859] RIP: 0033:0x7f83e5f5c10b [ 2344.802889] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2344.821891] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2344.828562] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2344.838546] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2344.844946] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2344.856603] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2344.867632] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2360.056043] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 2360.060091] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2360.093465] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2360.106182] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2360.116882] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2360.126612] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2360.157345] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2360.168342] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2360.176939] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2360.182725] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2360.190821] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2360.195266] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2360.203573] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2360.215695] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2360.222852] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2360.232405] Call Trace: [ 2360.236181] [ 2360.239737] ? show_regs.cold.9+0x22/0x2f [ 2360.244740] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2360.249815] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2360.252063] ? watchdog+0x40/0x40 [ 2360.256250] ? hrtimer_interrupt+0x128/0x2c0 [ 2360.260128] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2360.266971] ? apic_timer_interrupt+0xf/0x20 [ 2360.273860] [ 2360.277353] ? native_safe_halt+0x2e/0x50 [ 2360.279452] kvm_wait+0xcd/0xe0 [ 2360.284191] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2360.292137] do_raw_spin_lock+0x177/0x190 [ 2360.300139] _raw_spin_lock+0x21/0x30 [ 2360.307549] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2360.317680] obd_mod_free+0x46/0x140 [ptlrpc] [ 2360.327308] mdc_commit_open+0x91/0xb0 [mdc] [ 2360.330442] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2360.336148] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2360.348100] after_reply+0xe2c/0x1780 [ptlrpc] [ 2360.371796] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2360.382455] ? __next_timer_interrupt+0x160/0x160 [ 2360.389927] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2360.406469] ? do_wait_intr+0xf0/0xf0 [ 2360.412696] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2360.418567] kthread+0x1d1/0x200 [ 2360.424887] ? set_kthread_struct+0x70/0x70 [ 2360.427840] ret_from_fork+0x1f/0x30 [ 2372.063993] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 2372.065579] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 2372.074213] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2372.074292] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2372.074300] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2372.074302] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2372.074316] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2372.074320] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2372.074324] RAX: 0000000000004499 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2372.074327] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2372.074328] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2372.074330] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2372.074332] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2372.074334] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2372.074337] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2372.074339] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2372.074347] Call Trace: [ 2372.074352] [ 2372.074357] ? show_regs.cold.9+0x22/0x2f [ 2372.074365] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2372.074371] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2372.074376] ? watchdog+0x40/0x40 [ 2372.074380] ? hrtimer_interrupt+0x128/0x2c0 [ 2372.074384] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2372.074392] ? apic_timer_interrupt+0xf/0x20 [ 2372.074397] [ 2372.074401] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2372.074406] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2372.074411] do_raw_spin_lock+0x177/0x190 [ 2372.074416] _raw_spin_lock+0x21/0x30 [ 2372.074421] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2372.085332] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2372.085412] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2372.085419] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2372.085425] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2372.086156] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2372.086168] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2372.086173] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2372.086177] RAX: 0000000000001e91 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2372.086179] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2372.086181] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2372.086183] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2372.086185] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2372.086189] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2372.086191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2372.086194] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2372.086200] Call Trace: [ 2372.086206] [ 2372.086212] ? show_regs.cold.9+0x22/0x2f [ 2372.086221] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2372.086227] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2372.086231] ? watchdog+0x40/0x40 [ 2372.086236] ? hrtimer_interrupt+0x128/0x2c0 [ 2372.086240] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2372.086247] ? apic_timer_interrupt+0xf/0x20 [ 2372.086252] [ 2372.086255] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2372.086260] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2372.086265] do_raw_spin_lock+0x177/0x190 [ 2372.086270] _raw_spin_lock+0x21/0x30 [ 2372.086275] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2372.086478] process_one_work+0x2c8/0x700 [ 2372.086485] worker_thread+0x296/0x6e0 [ 2372.086502] ? rescuer_thread+0x570/0x570 [ 2372.086506] kthread+0x1d1/0x200 [ 2372.086510] ? set_kthread_struct+0x70/0x70 [ 2372.086513] ret_from_fork+0x1f/0x30 [ 2372.569631] ? do_raw_spin_unlock+0x75/0x190 [ 2372.572097] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2372.575430] ? do_wait_intr+0xf0/0xf0 [ 2372.577104] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2372.583774] mdc_reint+0x2d/0x110 [mdc] [ 2372.586054] mdc_unlink+0x658/0x870 [mdc] [ 2372.588464] lmv_unlink+0x34d/0xbd0 [lmv] [ 2372.590735] ll_unlink+0x24d/0x7f0 [lustre] [ 2372.594917] vfs_unlink+0x202/0x330 [ 2372.597349] do_unlinkat+0x2fa/0x460 [ 2372.599627] __x64_sys_unlink+0x37/0x50 [ 2372.601656] do_syscall_64+0xc1/0x440 [ 2372.605466] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2372.609138] RIP: 0033:0x7f83e5f5c10b [ 2372.615699] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2372.632431] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2372.636341] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2372.639781] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2372.644414] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2372.648261] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2372.655423] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2388.060971] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 2388.065030] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2388.101003] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2388.106899] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2388.112965] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2388.125516] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2388.134508] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2388.139258] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2388.144502] RDX: 000000000000022d RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2388.150592] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2388.159177] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2388.166849] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2388.177322] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2388.190267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2388.202981] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2388.206434] Call Trace: [ 2388.207683] [ 2388.213253] ? show_regs.cold.9+0x22/0x2f [ 2388.216431] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2388.224530] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2388.229283] ? watchdog+0x40/0x40 [ 2388.230493] ? hrtimer_interrupt+0x128/0x2c0 [ 2388.232498] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2388.235956] ? apic_timer_interrupt+0xf/0x20 [ 2388.237496] [ 2388.239293] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2388.242647] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2388.248361] do_raw_spin_lock+0x177/0x190 [ 2388.250505] _raw_spin_lock+0x21/0x30 [ 2388.252426] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2388.255188] obd_mod_free+0x46/0x140 [ptlrpc] [ 2388.257409] mdc_commit_open+0x91/0xb0 [mdc] [ 2388.259595] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2388.265779] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2388.269698] after_reply+0xe2c/0x1780 [ptlrpc] [ 2388.283040] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2388.285039] ? __next_timer_interrupt+0x160/0x160 [ 2388.296480] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2388.302678] ? do_wait_intr+0xf0/0xf0 [ 2388.305196] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2388.310397] kthread+0x1d1/0x200 [ 2388.314096] ? set_kthread_struct+0x70/0x70 [ 2388.321150] ret_from_fork+0x1f/0x30 [ 2400.060095] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 2400.063312] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2400.064103] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 2400.097108] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2400.097119] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2400.097124] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2400.103200] Modules linked in: lustre(O) [ 2400.111580] [ 2400.111590] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2400.111609] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2400.111613] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2400.111618] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2400.111621] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2400.111623] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2400.111625] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2400.111627] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2400.111631] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2400.111633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2400.111636] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2400.111643] Call Trace: [ 2400.111651] [ 2400.111659] ? show_regs.cold.9+0x22/0x2f [ 2400.111679] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2400.111689] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2400.111694] ? watchdog+0x40/0x40 [ 2400.111698] ? hrtimer_interrupt+0x128/0x2c0 [ 2400.111702] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2400.111708] ? apic_timer_interrupt+0xf/0x20 [ 2400.111713] [ 2400.111716] ? native_safe_halt+0x2e/0x50 [ 2400.111719] kvm_wait+0xcd/0xe0 [ 2400.111726] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2400.111737] do_raw_spin_lock+0x177/0x190 [ 2400.111742] _raw_spin_lock+0x21/0x30 [ 2400.111745] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2400.112501] process_one_work+0x2c8/0x700 [ 2400.112509] worker_thread+0x296/0x6e0 [ 2400.112513] ? rescuer_thread+0x570/0x570 [ 2400.112517] kthread+0x1d1/0x200 [ 2400.112521] ? set_kthread_struct+0x70/0x70 [ 2400.112525] ret_from_fork+0x1f/0x30 [ 2400.245711] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2400.277815] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2400.282555] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2400.290966] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2400.292601] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2400.303079] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2400.306310] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2400.310923] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2400.315385] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2400.318239] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2400.320754] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2400.323433] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2400.326745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2400.329129] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2400.332012] Call Trace: [ 2400.333041] [ 2400.333926] ? show_regs.cold.9+0x22/0x2f [ 2400.335603] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2400.337617] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2400.339406] ? watchdog+0x40/0x40 [ 2400.340818] ? hrtimer_interrupt+0x128/0x2c0 [ 2400.342687] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2400.344743] ? apic_timer_interrupt+0xf/0x20 [ 2400.346490] [ 2400.347398] ? native_safe_halt+0x2e/0x50 [ 2400.349160] kvm_wait+0xcd/0xe0 [ 2400.350492] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2400.352745] do_raw_spin_lock+0x177/0x190 [ 2400.354645] _raw_spin_lock+0x21/0x30 [ 2400.356087] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2400.358426] ? do_raw_spin_unlock+0x75/0x190 [ 2400.360290] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2400.362929] ? do_wait_intr+0xf0/0xf0 [ 2400.364445] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2400.366749] mdc_reint+0x2d/0x110 [mdc] [ 2400.368202] mdc_unlink+0x658/0x870 [mdc] [ 2400.369615] lmv_unlink+0x34d/0xbd0 [lmv] [ 2400.371003] ll_unlink+0x24d/0x7f0 [lustre] [ 2400.374070] vfs_unlink+0x202/0x330 [ 2400.375396] do_unlinkat+0x2fa/0x460 [ 2400.377788] __x64_sys_unlink+0x37/0x50 [ 2400.379749] do_syscall_64+0xc1/0x440 [ 2400.383161] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2400.388791] RIP: 0033:0x7f83e5f5c10b [ 2400.390910] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2400.401687] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2400.406110] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2400.410300] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2400.413828] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2400.417755] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2400.421832] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2400.968030] rcu: INFO: rcu_sched self-detected stall on CPU [ 2400.970507] rcu: 0-....: (274876 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=99640 [ 2400.976553] (t=420015 jiffies g=262441 q=56654) [ 2400.984083] NMI backtrace for cpu 0 [ 2400.989383] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2401.003212] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2401.009118] Call Trace: [ 2401.010211] [ 2401.012477] ? dump_stack+0xbb/0x10e [ 2401.016856] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2401.024777] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2401.030376] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2401.037515] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2401.046087] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2401.052702] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2401.058918] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2401.064554] ? update_process_times+0x6d/0xc0 [ 2401.067279] ? tick_sched_handle.isra.18+0x27/0x90 [ 2401.071917] ? tick_sched_timer+0x53/0xb0 [ 2401.074853] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2401.082776] ? tick_sched_handle.isra.18+0x90/0x90 [ 2401.087655] ? hrtimer_interrupt+0x128/0x2c0 [ 2401.090792] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2401.093613] ? apic_timer_interrupt+0xf/0x20 [ 2401.096139] [ 2401.097224] ? native_safe_halt+0x2e/0x50 [ 2401.099276] ? kvm_wait+0xcd/0xe0 [ 2401.100754] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2401.103274] ? do_raw_spin_lock+0x177/0x190 [ 2401.105223] ? _raw_spin_lock+0x21/0x30 [ 2401.106940] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2401.109829] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2401.112042] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2401.114299] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2401.117131] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2401.120171] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2401.123277] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2401.126305] ? __next_timer_interrupt+0x160/0x160 [ 2401.129265] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2401.131767] ? do_wait_intr+0xf0/0xf0 [ 2401.133946] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2401.137277] ? kthread+0x1d1/0x200 [ 2401.138782] ? set_kthread_struct+0x70/0x70 [ 2401.140945] ? ret_from_fork+0x1f/0x30 [ 2428.060687] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 2428.061204] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 2428.062014] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 2428.062078] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2428.062156] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2428.062162] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2428.062165] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2428.062181] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2428.062185] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2428.062189] RAX: 0000000000004c84 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2428.062191] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2428.062193] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2428.062370] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2428.062373] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2428.062377] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2428.062379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2428.062382] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2428.062390] Call Trace: [ 2428.062409] [ 2428.062414] ? show_regs.cold.9+0x22/0x2f [ 2428.062423] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2428.062429] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2428.062433] ? watchdog+0x40/0x40 [ 2428.062437] ? hrtimer_interrupt+0x128/0x2c0 [ 2428.062441] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2428.062450] ? apic_timer_interrupt+0xf/0x20 [ 2428.062454] [ 2428.062457] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2428.062462] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2428.062467] do_raw_spin_lock+0x177/0x190 [ 2428.062472] _raw_spin_lock+0x21/0x30 [ 2428.062478] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2428.062786] ? do_raw_spin_unlock+0x75/0x190 [ 2428.062793] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2428.063467] ? do_wait_intr+0xf0/0xf0 [ 2428.063477] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2428.063685] mdc_reint+0x2d/0x110 [mdc] [ 2428.063721] mdc_unlink+0x658/0x870 [mdc] [ 2428.063745] lmv_unlink+0x34d/0xbd0 [lmv] [ 2428.063776] ll_unlink+0x24d/0x7f0 [lustre] [ 2428.063903] vfs_unlink+0x202/0x330 [ 2428.063916] do_unlinkat+0x2fa/0x460 [ 2428.063921] __x64_sys_unlink+0x37/0x50 [ 2428.063925] do_syscall_64+0xc1/0x440 [ 2428.063934] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2428.063942] RIP: 0033:0x7f83e5f5c10b [ 2428.063950] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2428.063953] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2428.063958] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2428.063960] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2428.063962] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2428.063964] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2428.063966] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2428.068321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2428.068395] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2428.068412] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2428.068418] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2428.086036] Modules linked in: [ 2428.091045] [ 2428.091056] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2428.091077] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2428.091082] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2428.091161] RAX: 0000000000007d87 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2428.091165] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2428.091167] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2428.091169] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2428.091172] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2428.091175] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2428.091178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2428.091181] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2428.091254] Call Trace: [ 2428.091260] [ 2428.091265] ? show_regs.cold.9+0x22/0x2f [ 2428.091274] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2428.091280] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2428.091284] ? watchdog+0x40/0x40 [ 2428.091289] ? hrtimer_interrupt+0x128/0x2c0 [ 2428.091293] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2428.091300] ? apic_timer_interrupt+0xf/0x20 [ 2428.091304] [ 2428.091307] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2428.091311] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2428.091318] do_raw_spin_lock+0x177/0x190 [ 2428.091322] _raw_spin_lock+0x21/0x30 [ 2428.091326] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2428.092165] process_one_work+0x2c8/0x700 [ 2428.092173] worker_thread+0x296/0x6e0 [ 2428.092177] ? rescuer_thread+0x570/0x570 [ 2428.092181] kthread+0x1d1/0x200 [ 2428.092185] ? set_kthread_struct+0x70/0x70 [ 2428.092254] ret_from_fork+0x1f/0x30 [ 2428.554532] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2428.583567] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2428.591559] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2428.598881] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2428.605780] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2428.618297] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2428.621587] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2428.636443] RDX: 000000000000791f RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2428.643704] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2428.648073] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2428.654482] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2428.659870] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2428.666024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2428.670539] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2428.675767] Call Trace: [ 2428.677586] [ 2428.680109] ? show_regs.cold.9+0x22/0x2f [ 2428.682881] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2428.685689] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2428.690111] ? watchdog+0x40/0x40 [ 2428.694983] ? hrtimer_interrupt+0x128/0x2c0 [ 2428.697672] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2428.699299] ? apic_timer_interrupt+0xf/0x20 [ 2428.703258] [ 2428.704875] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2428.708429] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2428.711572] do_raw_spin_lock+0x177/0x190 [ 2428.719834] _raw_spin_lock+0x21/0x30 [ 2428.724549] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2428.731458] obd_mod_free+0x46/0x140 [ptlrpc] [ 2428.736084] mdc_commit_open+0x91/0xb0 [mdc] [ 2428.739815] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2428.744194] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2428.747465] after_reply+0xe2c/0x1780 [ptlrpc] [ 2428.755462] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2428.759517] ? __next_timer_interrupt+0x160/0x160 [ 2428.763444] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2428.767816] ? do_wait_intr+0xf0/0xf0 [ 2428.769458] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2428.773844] kthread+0x1d1/0x200 [ 2428.775386] ? set_kthread_struct+0x70/0x70 [ 2428.779434] ret_from_fork+0x1f/0x30 [ 2456.056040] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 2456.060088] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 2456.060599] Modules linked in: [ 2456.062203] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2456.062213] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2456.062307] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2456.062312] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2456.062315] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2456.062331] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2456.062344] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2456.062348] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2456.062350] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2456.062353] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2456.062354] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2456.062357] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2456.062360] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2456.062362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2456.062369] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2456.062376] Call Trace: [ 2456.062382] [ 2456.062387] ? show_regs.cold.9+0x22/0x2f [ 2456.062396] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2456.062409] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2456.062414] ? watchdog+0x40/0x40 [ 2456.062418] ? hrtimer_interrupt+0x128/0x2c0 [ 2456.062422] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2456.062428] ? apic_timer_interrupt+0xf/0x20 [ 2456.062433] [ 2456.062440] ? native_safe_halt+0x2e/0x50 [ 2456.062445] kvm_wait+0xcd/0xe0 [ 2456.062452] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2456.062459] do_raw_spin_lock+0x177/0x190 [ 2456.062463] _raw_spin_lock+0x21/0x30 [ 2456.062468] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2456.062774] ? do_raw_spin_unlock+0x75/0x190 [ 2456.062780] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2456.063096] ? do_wait_intr+0xf0/0xf0 [ 2456.063104] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2456.063600] mdc_reint+0x2d/0x110 [mdc] [ 2456.063654] mdc_unlink+0x658/0x870 [mdc] [ 2456.063695] lmv_unlink+0x34d/0xbd0 [lmv] [ 2456.063736] ll_unlink+0x24d/0x7f0 [lustre] [ 2456.063954] vfs_unlink+0x202/0x330 [ 2456.063963] do_unlinkat+0x2fa/0x460 [ 2456.063969] __x64_sys_unlink+0x37/0x50 [ 2456.063972] do_syscall_64+0xc1/0x440 [ 2456.063980] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2456.063992] RIP: 0033:0x7f83e5f5c10b [ 2456.064009] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2456.064013] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2456.064078] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2456.064080] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2456.064082] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2456.064084] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2456.064086] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2456.066180] Modules linked in: [ 2456.068097] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) [ 2456.071626] lustre(O) [ 2456.111340] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2456.111382] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2456.111388] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2456.111391] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2456.111403] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2456.111406] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2456.111410] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2456.111413] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2456.111415] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2456.111417] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2456.111419] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2456.111421] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2456.111424] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2456.111426] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2456.111435] Call Trace: [ 2456.111450] [ 2456.111456] ? show_regs.cold.9+0x22/0x2f [ 2456.111465] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2456.111471] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2456.111475] ? watchdog+0x40/0x40 [ 2456.111479] ? hrtimer_interrupt+0x128/0x2c0 [ 2456.111482] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2456.111488] ? apic_timer_interrupt+0xf/0x20 [ 2456.111493] [ 2456.111496] ? native_safe_halt+0x2e/0x50 [ 2456.111500] kvm_wait+0xcd/0xe0 [ 2456.111507] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2456.111514] do_raw_spin_lock+0x177/0x190 [ 2456.111518] _raw_spin_lock+0x21/0x30 [ 2456.111523] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2456.111785] obd_mod_free+0x46/0x140 [ptlrpc] [ 2456.111993] mdc_commit_open+0x91/0xb0 [mdc] [ 2456.112028] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2456.125099] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2456.125189] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2456.125195] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2456.125202] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2456.134289] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2456.138801] [ 2456.138810] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2456.138826] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2456.138841] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2456.138847] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2456.138850] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2456.138852] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2456.138854] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2456.138856] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2456.138860] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2456.138862] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2456.138864] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2456.138872] Call Trace: [ 2456.138877] [ 2456.138883] ? show_regs.cold.9+0x22/0x2f [ 2456.150536] after_reply+0xe2c/0x1780 [ptlrpc] [ 2456.150859] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2456.151062] ? __next_timer_interrupt+0x160/0x160 [ 2456.151072] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2456.151287] ? do_wait_intr+0xf0/0xf0 [ 2456.151296] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2456.154395] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2456.159548] kthread+0x1d1/0x200 [ 2456.165174] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2456.169544] ? set_kthread_struct+0x70/0x70 [ 2456.173074] ? watchdog+0x40/0x40 [ 2456.173093] ? hrtimer_interrupt+0x128/0x2c0 [ 2456.173099] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2456.173108] ? apic_timer_interrupt+0xf/0x20 [ 2456.173112] [ 2456.173116] ? native_safe_halt+0x2e/0x50 [ 2456.173120] kvm_wait+0xcd/0xe0 [ 2456.173139] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2456.173148] do_raw_spin_lock+0x177/0x190 [ 2456.173152] _raw_spin_lock+0x21/0x30 [ 2456.173156] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2456.179356] ret_from_fork+0x1f/0x30 [ 2456.694822] process_one_work+0x2c8/0x700 [ 2456.696708] worker_thread+0x296/0x6e0 [ 2456.697917] ? rescuer_thread+0x570/0x570 [ 2456.701833] kthread+0x1d1/0x200 [ 2456.704075] ? set_kthread_struct+0x70/0x70 [ 2456.709559] ret_from_fork+0x1f/0x30 [ 2484.056554] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 2484.060023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 2484.065909] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2484.065923] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2484.066125] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2484.066132] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2484.066135] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2484.066149] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2484.066153] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2484.066157] RAX: 0000000000007f85 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2484.066160] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2484.066161] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2484.066163] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2484.066166] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2484.066169] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2484.066172] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2484.066174] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2484.066177] Modules linked in: lustre(O) [ 2484.066184] Call Trace: [ 2484.066184] osp(O) ofd(O) lod(O) mdt(O) [ 2484.066193] [ 2484.066194] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) [ 2484.066203] ? show_regs.cold.9+0x22/0x2f [ 2484.066211] lmv(O) ec(O) fid(O) fld(O) [ 2484.066218] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2484.066245] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2484.066249] ? watchdog+0x40/0x40 [ 2484.066254] ? hrtimer_interrupt+0x128/0x2c0 [ 2484.066258] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2484.066268] ? apic_timer_interrupt+0xf/0x20 [ 2484.066273] [ 2484.066276] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2484.066281] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2484.066285] do_raw_spin_lock+0x177/0x190 [ 2484.066290] _raw_spin_lock+0x21/0x30 [ 2484.066297] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2484.066576] ? do_raw_spin_unlock+0x75/0x190 [ 2484.066581] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2484.069924] ? do_wait_intr+0xf0/0xf0 [ 2484.069941] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2484.070192] mdc_reint+0x2d/0x110 [mdc] [ 2484.070292] mdc_unlink+0x658/0x870 [mdc] [ 2484.070317] lmv_unlink+0x34d/0xbd0 [lmv] [ 2484.070605] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2484.070679] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2484.070685] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2484.070692] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2484.070350] ll_unlink+0x24d/0x7f0 [lustre] [ 2484.071111] vfs_unlink+0x202/0x330 [ 2484.071119] do_unlinkat+0x2fa/0x460 [ 2484.071124] __x64_sys_unlink+0x37/0x50 [ 2484.071128] do_syscall_64+0xc1/0x440 [ 2484.071135] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2484.071143] RIP: 0033:0x7f83e5f5c10b [ 2484.071150] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2484.071154] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2484.071158] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2484.071160] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2484.071163] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2484.071164] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2484.071166] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2484.081085] ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2484.081148] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2484.100464] [ 2484.100474] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2484.100490] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2484.100495] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2484.100499] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2484.100502] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2484.100504] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2484.100506] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2484.100508] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2484.100511] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2484.100514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2484.100516] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2484.100524] Call Trace: [ 2484.100529] [ 2484.100534] ? show_regs.cold.9+0x22/0x2f [ 2484.100544] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2484.100550] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2484.100554] ? watchdog+0x40/0x40 [ 2484.100558] ? hrtimer_interrupt+0x128/0x2c0 [ 2484.100562] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2484.100568] ? apic_timer_interrupt+0xf/0x20 [ 2484.100572] [ 2484.100575] ? native_safe_halt+0x2e/0x50 [ 2484.100579] kvm_wait+0xcd/0xe0 [ 2484.100586] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2484.100596] do_raw_spin_lock+0x177/0x190 [ 2484.100601] _raw_spin_lock+0x21/0x30 [ 2484.100605] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2484.101988] process_one_work+0x2c8/0x700 [ 2484.101999] worker_thread+0x296/0x6e0 [ 2484.102003] ? rescuer_thread+0x570/0x570 [ 2484.102007] kthread+0x1d1/0x200 [ 2484.102012] ? set_kthread_struct+0x70/0x70 [ 2484.102015] ret_from_fork+0x1f/0x30 [ 2484.650118] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2484.660313] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2484.666083] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2484.680997] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2484.684682] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2484.688324] RDX: 000000000000432c RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2484.691076] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2484.693911] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2484.696615] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2484.699307] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2484.702299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2484.707178] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2484.715280] Call Trace: [ 2484.716832] [ 2484.721153] ? show_regs.cold.9+0x22/0x2f [ 2484.722861] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2484.724557] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2484.726824] ? watchdog+0x40/0x40 [ 2484.728211] ? hrtimer_interrupt+0x128/0x2c0 [ 2484.730532] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2484.732567] ? apic_timer_interrupt+0xf/0x20 [ 2484.734846] [ 2484.735757] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2484.738031] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2484.740434] do_raw_spin_lock+0x177/0x190 [ 2484.742017] _raw_spin_lock+0x21/0x30 [ 2484.743486] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2484.745742] obd_mod_free+0x46/0x140 [ptlrpc] [ 2484.749790] mdc_commit_open+0x91/0xb0 [mdc] [ 2484.751453] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2484.753280] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2484.755215] after_reply+0xe2c/0x1780 [ptlrpc] [ 2484.757426] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2484.760060] ? __next_timer_interrupt+0x160/0x160 [ 2484.761812] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2484.764070] ? do_wait_intr+0xf0/0xf0 [ 2484.766541] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2484.769371] kthread+0x1d1/0x200 [ 2484.770625] ? set_kthread_struct+0x70/0x70 [ 2484.771904] ret_from_fork+0x1f/0x30 [ 2512.056053] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 2512.059025] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2512.060025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 2512.062032] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2512.062043] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2512.062111] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2512.062115] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2512.062118] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2512.062131] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2512.062135] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2512.062139] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2512.062141] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2512.062143] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2512.062145] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2512.062148] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2512.062151] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2512.062154] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2512.062156] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2512.062164] Call Trace: [ 2512.062168] [ 2512.062173] ? show_regs.cold.9+0x22/0x2f [ 2512.062182] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2512.062187] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2512.062191] ? watchdog+0x40/0x40 [ 2512.062195] ? hrtimer_interrupt+0x128/0x2c0 [ 2512.062199] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2512.062204] ? apic_timer_interrupt+0xf/0x20 [ 2512.062208] [ 2512.062211] ? native_safe_halt+0x2e/0x50 [ 2512.062219] kvm_wait+0xcd/0xe0 [ 2512.062226] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2512.062232] do_raw_spin_lock+0x177/0x190 [ 2512.062236] _raw_spin_lock+0x21/0x30 [ 2512.062240] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2512.062569] ? do_raw_spin_unlock+0x75/0x190 [ 2512.062574] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2512.062778] ? do_wait_intr+0xf0/0xf0 [ 2512.062785] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2512.062985] mdc_reint+0x2d/0x110 [mdc] [ 2512.063021] mdc_unlink+0x658/0x870 [mdc] [ 2512.063048] lmv_unlink+0x34d/0xbd0 [lmv] [ 2512.063078] ll_unlink+0x24d/0x7f0 [lustre] [ 2512.063190] vfs_unlink+0x202/0x330 [ 2512.063197] do_unlinkat+0x2fa/0x460 [ 2512.063202] __x64_sys_unlink+0x37/0x50 [ 2512.063205] do_syscall_64+0xc1/0x440 [ 2512.063211] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2512.063217] RIP: 0033:0x7f83e5f5c10b [ 2512.063222] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2512.063225] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2512.063229] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2512.063232] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2512.063233] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2512.063236] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2512.063238] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2512.079316] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2512.082945] Modules linked in: [ 2512.085780] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2512.085790] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2512.085807] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2512.085811] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2512.085818] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2512.085820] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2512.085822] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2512.085824] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2512.085826] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2512.085829] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2512.085831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2512.085834] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2512.085841] Call Trace: [ 2512.085847] [ 2512.085852] ? show_regs.cold.9+0x22/0x2f [ 2512.085863] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2512.085869] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2512.085873] ? watchdog+0x40/0x40 [ 2512.085878] ? hrtimer_interrupt+0x128/0x2c0 [ 2512.085882] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2512.085889] ? apic_timer_interrupt+0xf/0x20 [ 2512.085893] [ 2512.085896] ? native_safe_halt+0x2e/0x50 [ 2512.085901] kvm_wait+0xcd/0xe0 [ 2512.085909] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2512.085916] do_raw_spin_lock+0x177/0x190 [ 2512.085920] _raw_spin_lock+0x21/0x30 [ 2512.085926] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2512.086311] obd_mod_free+0x46/0x140 [ptlrpc] [ 2512.086683] mdc_commit_open+0x91/0xb0 [mdc] [ 2512.086725] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2512.086984] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2512.087230] after_reply+0xe2c/0x1780 [ptlrpc] [ 2512.087421] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2512.087705] ? __next_timer_interrupt+0x160/0x160 [ 2512.087714] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2512.088057] ? do_wait_intr+0xf0/0xf0 [ 2512.088068] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2512.088266] kthread+0x1d1/0x200 [ 2512.088272] ? set_kthread_struct+0x70/0x70 [ 2512.088275] ret_from_fork+0x1f/0x30 [ 2512.483086] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2512.526655] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2512.534253] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2512.540765] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2512.544656] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2512.549346] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2512.563388] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2512.571477] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2512.581102] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2512.582904] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2512.593144] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2512.599973] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2512.612365] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2512.620300] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2512.621781] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2512.627123] Call Trace: [ 2512.627977] [ 2512.628479] ? show_regs.cold.9+0x22/0x2f [ 2512.631483] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2512.634078] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2512.635920] ? watchdog+0x40/0x40 [ 2512.636880] ? hrtimer_interrupt+0x128/0x2c0 [ 2512.645340] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2512.648794] ? apic_timer_interrupt+0xf/0x20 [ 2512.650077] [ 2512.666844] ? native_safe_halt+0x2e/0x50 [ 2512.669202] kvm_wait+0xcd/0xe0 [ 2512.683521] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2512.687479] do_raw_spin_lock+0x177/0x190 [ 2512.697903] _raw_spin_lock+0x21/0x30 [ 2512.700613] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2512.702339] process_one_work+0x2c8/0x700 [ 2512.719385] worker_thread+0x296/0x6e0 [ 2512.722544] ? rescuer_thread+0x570/0x570 [ 2512.724380] kthread+0x1d1/0x200 [ 2512.729128] ? set_kthread_struct+0x70/0x70 [ 2512.733474] ret_from_fork+0x1f/0x30 [ 2532.064671] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 2532.076429] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2532.181834] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2532.190181] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2532.194582] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2532.196705] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2532.208427] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2532.211558] RAX: 0000000000004e1d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2532.214025] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 2532.219354] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2532.230845] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 2532.235744] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 2532.239998] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 2532.243576] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2532.246575] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 2532.249062] Call Trace: [ 2532.250028] [ 2532.251026] ? show_regs.cold.9+0x22/0x2f [ 2532.254337] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2532.258758] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2532.261062] ? watchdog+0x40/0x40 [ 2532.263037] ? hrtimer_interrupt+0x128/0x2c0 [ 2532.268633] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2532.270339] ? apic_timer_interrupt+0xf/0x20 [ 2532.272570] [ 2532.274853] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2532.279262] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2532.283497] do_raw_spin_lock+0x177/0x190 [ 2532.285555] _raw_spin_lock+0x21/0x30 [ 2532.288060] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2532.291976] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2532.298151] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2532.302478] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2532.305877] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2532.308684] ? woken_wake_function+0x30/0x30 [ 2532.310991] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2532.313875] kthread+0x1d1/0x200 [ 2532.315371] ? set_kthread_struct+0x70/0x70 [ 2532.319033] ret_from_fork+0x1f/0x30 [ 2540.056024] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 2540.061375] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2540.063043] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2540.063100] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 2540.063106] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2540.063169] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2540.063175] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2540.063181] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2540.063714] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2540.063725] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2540.063729] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2540.063733] RAX: 0000000000006303 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2540.063735] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2540.063738] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2540.063740] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2540.063742] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2540.063745] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2540.063748] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2540.063751] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2540.063759] Call Trace: [ 2540.063764] [ 2540.063769] ? show_regs.cold.9+0x22/0x2f [ 2540.063777] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2540.063783] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2540.063787] ? watchdog+0x40/0x40 [ 2540.063791] ? hrtimer_interrupt+0x128/0x2c0 [ 2540.063795] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2540.063802] ? apic_timer_interrupt+0xf/0x20 [ 2540.063807] [ 2540.063810] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2540.063815] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2540.063820] do_raw_spin_lock+0x177/0x190 [ 2540.063824] _raw_spin_lock+0x21/0x30 [ 2540.063829] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2540.064251] process_one_work+0x2c8/0x700 [ 2540.064258] worker_thread+0x296/0x6e0 [ 2540.064262] ? rescuer_thread+0x570/0x570 [ 2540.064266] kthread+0x1d1/0x200 [ 2540.064271] ? set_kthread_struct+0x70/0x70 [ 2540.064274] ret_from_fork+0x1f/0x30 [ 2540.090738] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2540.090752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2540.090756] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2540.090772] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2540.090776] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2540.090783] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2540.090785] RDX: 0000000000000680 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2540.090787] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2540.090789] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2540.090791] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2540.090794] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2540.090797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2540.090799] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2540.090816] Call Trace: [ 2540.090822] [ 2540.090829] ? show_regs.cold.9+0x22/0x2f [ 2540.090839] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2540.090845] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2540.090849] ? watchdog+0x40/0x40 [ 2540.090854] ? hrtimer_interrupt+0x128/0x2c0 [ 2540.090859] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2540.090867] ? apic_timer_interrupt+0xf/0x20 [ 2540.090872] [ 2540.090874] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2540.090879] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2540.090884] do_raw_spin_lock+0x177/0x190 [ 2540.090889] _raw_spin_lock+0x21/0x30 [ 2540.090897] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2540.095978] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2540.096057] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2540.096065] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2540.096067] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2540.096080] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2540.096084] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2540.096089] RAX: 00000000000037f7 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2540.096091] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2540.096093] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2540.096095] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2540.096097] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2540.096101] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2540.096104] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2540.096106] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2540.096115] Call Trace: [ 2540.096120] [ 2540.096125] ? show_regs.cold.9+0x22/0x2f [ 2540.096134] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2540.096152] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2540.096163] ? watchdog+0x40/0x40 [ 2540.096167] ? hrtimer_interrupt+0x128/0x2c0 [ 2540.096171] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2540.096180] ? apic_timer_interrupt+0xf/0x20 [ 2540.096184] [ 2540.096187] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2540.096192] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2540.096197] do_raw_spin_lock+0x177/0x190 [ 2540.096201] _raw_spin_lock+0x21/0x30 [ 2540.096206] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2540.096728] ? do_raw_spin_unlock+0x75/0x190 [ 2540.096734] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2540.097118] ? do_wait_intr+0xf0/0xf0 [ 2540.097126] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2540.097354] mdc_reint+0x2d/0x110 [mdc] [ 2540.097395] mdc_unlink+0x658/0x870 [mdc] [ 2540.097420] lmv_unlink+0x34d/0xbd0 [lmv] [ 2540.097663] ll_unlink+0x24d/0x7f0 [lustre] [ 2540.097775] vfs_unlink+0x202/0x330 [ 2540.097783] do_unlinkat+0x2fa/0x460 [ 2540.097788] __x64_sys_unlink+0x37/0x50 [ 2540.097791] do_syscall_64+0xc1/0x440 [ 2540.097798] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2540.097817] RIP: 0033:0x7f83e5f5c10b [ 2540.097823] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2540.097828] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2540.097833] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2540.097835] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2540.097836] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2540.097838] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2540.097840] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2540.634064] obd_mod_free+0x46/0x140 [ptlrpc] [ 2540.637124] mdc_commit_open+0x91/0xb0 [mdc] [ 2540.643930] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2540.655475] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2540.661088] after_reply+0xe2c/0x1780 [ptlrpc] [ 2540.670190] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2540.681866] ? __next_timer_interrupt+0x160/0x160 [ 2540.689497] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2540.698192] ? do_wait_intr+0xf0/0xf0 [ 2540.699583] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2540.713679] kthread+0x1d1/0x200 [ 2540.715344] ? set_kthread_struct+0x70/0x70 [ 2540.725878] ret_from_fork+0x1f/0x30 [ 2545.372676] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 32s! [ 2545.375320] Showing busy workqueues and worker pools: [ 2545.377041] workqueue events: flags=0x0 [ 2545.378330] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2545.380359] pending: osc_grant_work_handler [osc], push_to_pool [ 2545.382358] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2545.386953] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 2545.397982] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2545.408347] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 2545.419135] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2545.429503] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 2545.440367] workqueue events_long: flags=0x0 [ 2545.449921] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2545.468315] pending: lu_objects_flush [obdclass] [ 2545.470123] workqueue events_unbound: flags=0x2 [ 2545.475235] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/512 refcnt=3 [ 2545.484928] pending: flush_memcg_stats_dwork [ 2545.490282] workqueue events_power_efficient: flags=0x80 [ 2545.502535] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2545.506369] pending: neigh_periodic_work, do_cache_clean [ 2545.518384] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2545.534910] pending: neigh_periodic_work [ 2545.540635] workqueue mm_percpu_wq: flags=0x8 [ 2545.549234] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2545.552949] pending: vmstat_update, lru_add_drain_per_cpu [ 2545.573374] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2545.584836] pending: vmstat_update [ 2545.586207] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2545.591360] pending: lru_add_drain_per_cpu BAR(48) [ 2545.600922] workqueue ipv6_addrconf: flags=0x40008 [ 2545.604415] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2545.612009] pending: addrconf_verify_work [ 2545.617269] workqueue knbd0-recv: flags=0x1a [ 2545.619010] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2545.629923] in-flight: 650:recv_work [ 2545.641980] workqueue ptlrpc_pinger: flags=0x2 [ 2545.648579] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2545.654676] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2545.661990] pool 8: cpus=0-3 flags=0x4 nice=0 hung=32s workers=4 idle: 36 466 29757 [ 2545.670836] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2535s workers=2 idle: 652 [ 2560.063303] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 2560.073819] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2560.137469] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2560.143096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2560.149801] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2560.159248] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2560.183563] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2560.187082] RAX: 0000000000007f10 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2560.190320] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 2560.193037] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2560.195572] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 2560.197501] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 2560.199717] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 2560.202280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2560.203796] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 2560.205829] Call Trace: [ 2560.206565] [ 2560.207095] ? show_regs.cold.9+0x22/0x2f [ 2560.208134] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2560.209476] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2560.210464] ? watchdog+0x40/0x40 [ 2560.211230] ? hrtimer_interrupt+0x128/0x2c0 [ 2560.212110] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2560.213322] ? apic_timer_interrupt+0xf/0x20 [ 2560.214698] [ 2560.215211] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2560.216604] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2560.218390] do_raw_spin_lock+0x177/0x190 [ 2560.219938] _raw_spin_lock+0x21/0x30 [ 2560.221368] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2560.228470] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2560.232388] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2560.235750] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2560.241792] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2560.249232] ? woken_wake_function+0x30/0x30 [ 2560.253344] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2560.255284] kthread+0x1d1/0x200 [ 2560.258884] ? set_kthread_struct+0x70/0x70 [ 2560.261501] ret_from_fork+0x1f/0x30 [ 2568.060019] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 2568.062413] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2568.063172] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2568.063302] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2568.063308] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2568.063316] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2568.063607] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2568.063618] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2568.063622] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2568.063627] RAX: 0000000000001a83 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2568.063629] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2568.063631] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2568.063633] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2568.063635] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2568.063638] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2568.063640] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2568.063642] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2568.063652] Call Trace: [ 2568.063658] [ 2568.063663] ? show_regs.cold.9+0x22/0x2f [ 2568.063673] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2568.063680] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2568.063684] ? watchdog+0x40/0x40 [ 2568.063689] ? hrtimer_interrupt+0x128/0x2c0 [ 2568.063693] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2568.063701] ? apic_timer_interrupt+0xf/0x20 [ 2568.063706] [ 2568.063709] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2568.063714] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2568.063719] do_raw_spin_lock+0x177/0x190 [ 2568.063725] _raw_spin_lock+0x21/0x30 [ 2568.063730] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2568.066088] process_one_work+0x2c8/0x700 [ 2568.066098] worker_thread+0x296/0x6e0 [ 2568.066102] ? rescuer_thread+0x570/0x570 [ 2568.066107] kthread+0x1d1/0x200 [ 2568.066111] ? set_kthread_struct+0x70/0x70 [ 2568.066115] ret_from_fork+0x1f/0x30 [ 2568.293493] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2568.313500] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2568.318435] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2568.322281] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2568.324088] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2568.330416] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2568.332441] RAX: 0000000000001b8f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2568.334434] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2568.337539] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2568.340118] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2568.343258] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2568.346477] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2568.349593] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2568.351735] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2568.354582] Call Trace: [ 2568.355825] [ 2568.356689] ? show_regs.cold.9+0x22/0x2f [ 2568.358378] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2568.360055] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2568.362163] ? watchdog+0x40/0x40 [ 2568.363641] ? hrtimer_interrupt+0x128/0x2c0 [ 2568.365362] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2568.367291] ? apic_timer_interrupt+0xf/0x20 [ 2568.368821] [ 2568.369696] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2568.371762] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2568.373809] do_raw_spin_lock+0x177/0x190 [ 2568.375492] _raw_spin_lock+0x21/0x30 [ 2568.376822] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2568.379352] ? do_raw_spin_unlock+0x75/0x190 [ 2568.381105] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2568.383384] ? do_wait_intr+0xf0/0xf0 [ 2568.384713] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2568.387116] mdc_reint+0x2d/0x110 [mdc] [ 2568.388646] mdc_unlink+0x658/0x870 [mdc] [ 2568.390080] lmv_unlink+0x34d/0xbd0 [lmv] [ 2568.391518] ll_unlink+0x24d/0x7f0 [lustre] [ 2568.393416] vfs_unlink+0x202/0x330 [ 2568.394671] do_unlinkat+0x2fa/0x460 [ 2568.396347] __x64_sys_unlink+0x37/0x50 [ 2568.398012] do_syscall_64+0xc1/0x440 [ 2568.399544] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2568.401618] RIP: 0033:0x7f83e5f5c10b [ 2568.403059] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2568.411000] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2568.414872] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2568.417458] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2568.420528] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2568.424086] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2568.426522] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2572.057878] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 2572.060475] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2572.078858] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2572.083193] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2572.086742] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2572.088112] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2572.092909] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2572.094638] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2572.096857] RDX: 0000000000001ab2 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2572.098903] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2572.101690] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2572.103419] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2572.105704] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2572.110592] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2572.114960] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2572.120864] Call Trace: [ 2572.121603] [ 2572.122082] ? show_regs.cold.9+0x22/0x2f [ 2572.123806] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2572.126636] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2572.128181] ? watchdog+0x40/0x40 [ 2572.128906] ? hrtimer_interrupt+0x128/0x2c0 [ 2572.129883] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2572.131500] ? apic_timer_interrupt+0xf/0x20 [ 2572.134030] [ 2572.136306] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2572.137585] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2572.138785] do_raw_spin_lock+0x177/0x190 [ 2572.141305] _raw_spin_lock+0x21/0x30 [ 2572.142816] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2572.144817] obd_mod_free+0x46/0x140 [ptlrpc] [ 2572.146076] mdc_commit_open+0x91/0xb0 [mdc] [ 2572.148647] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2572.151671] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2572.153739] after_reply+0xe2c/0x1780 [ptlrpc] [ 2572.156030] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2572.157766] ? __next_timer_interrupt+0x160/0x160 [ 2572.158953] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2572.160790] ? do_wait_intr+0xf0/0xf0 [ 2572.161977] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2572.164680] kthread+0x1d1/0x200 [ 2572.165875] ? set_kthread_struct+0x70/0x70 [ 2572.167465] ret_from_fork+0x1f/0x30 [ 2575.842029] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 62s! [ 2575.845511] Showing busy workqueues and worker pools: [ 2575.848084] workqueue events: flags=0x0 [ 2575.851601] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2575.855034] pending: osc_grant_work_handler [osc], push_to_pool [ 2575.858839] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2575.865560] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 2575.870321] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2575.873377] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 2575.876904] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2575.879895] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 2575.884096] workqueue events_long: flags=0x0 [ 2575.886481] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2575.889586] pending: lu_objects_flush [obdclass] [ 2575.892161] workqueue events_unbound: flags=0x2 [ 2575.894151] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2575.897493] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2575.900270] workqueue events_power_efficient: flags=0x80 [ 2575.902670] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2575.905882] pending: neigh_periodic_work, do_cache_clean [ 2575.908817] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2575.912375] pending: neigh_periodic_work [ 2575.914373] workqueue mm_percpu_wq: flags=0x8 [ 2575.916750] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2575.920087] pending: vmstat_update, lru_add_drain_per_cpu [ 2575.922686] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2575.926387] pending: vmstat_update [ 2575.928260] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2575.931434] pending: lru_add_drain_per_cpu BAR(48) [ 2575.933754] workqueue ipv6_addrconf: flags=0x40008 [ 2575.936074] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2575.938979] pending: addrconf_verify_work [ 2575.940786] workqueue knbd0-recv: flags=0x1a [ 2575.942853] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2575.946015] in-flight: 650:recv_work [ 2575.947720] workqueue ptlrpc_pinger: flags=0x2 [ 2575.949712] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2575.952682] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2575.954911] pool 8: cpus=0-3 flags=0x4 nice=0 hung=62s workers=4 idle: 36 466 29757 [ 2575.958611] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2566s workers=2 idle: 652 [ 2580.970128] rcu: INFO: rcu_sched self-detected stall on CPU [ 2580.973443] rcu: 0-....: (389043 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 2580.977552] (t=600018 jiffies g=262441 q=57098) [ 2580.979547] rcu: rcu_sched kthread starved for 69207 jiffies! g262441 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 2580.984639] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2580.989403] rcu: RCU grace-period kthread stack dump: [ 2580.991560] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2580.998419] Call Trace: [ 2580.998980] __schedule+0x351/0xcb0 [ 2580.999844] schedule+0xc0/0x180 [ 2581.001022] schedule_timeout+0xb4/0x190 [ 2581.002637] ? do_raw_spin_unlock+0x75/0x190 [ 2581.008788] ? __next_timer_interrupt+0x160/0x160 [ 2581.011085] ? prepare_to_swait_event+0xf3/0x140 [ 2581.017307] rcu_gp_kthread+0x8dd/0x15b0 [ 2581.019541] ? force_qs_rnp+0x2c0/0x2c0 [ 2581.022900] kthread+0x1d1/0x200 [ 2581.026260] ? set_kthread_struct+0x70/0x70 [ 2581.029260] ret_from_fork+0x1f/0x30 [ 2581.031292] rcu: Stack dump where RCU GP kthread last ran: [ 2581.035190] Sending NMI from CPU 0 to CPUs 1: [ 2581.036772] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 2581.038554] NMI backtrace for cpu 0 [ 2581.044465] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2581.057390] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2581.065323] Call Trace: [ 2581.067582] [ 2581.068789] ? dump_stack+0xbb/0x10e [ 2581.073308] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2581.076583] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2581.079743] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2581.082249] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2581.084781] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2581.087897] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2581.090588] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2581.093417] ? update_process_times+0x6d/0xc0 [ 2581.096326] ? tick_sched_handle.isra.18+0x27/0x90 [ 2581.098705] ? tick_sched_timer+0x53/0xb0 [ 2581.100889] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2581.105934] ? tick_sched_handle.isra.18+0x90/0x90 [ 2581.108591] ? hrtimer_interrupt+0x128/0x2c0 [ 2581.111528] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2581.116587] ? apic_timer_interrupt+0xf/0x20 [ 2581.120086] [ 2581.121676] ? native_safe_halt+0x2e/0x50 [ 2581.123795] ? kvm_wait+0xcd/0xe0 [ 2581.125830] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2581.128547] ? do_raw_spin_lock+0x177/0x190 [ 2581.131429] ? _raw_spin_lock+0x21/0x30 [ 2581.133688] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2581.136779] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2581.140404] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2581.143180] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2581.145868] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2581.148621] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2581.151122] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2581.153650] ? __next_timer_interrupt+0x160/0x160 [ 2581.156259] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2581.158900] ? do_wait_intr+0xf0/0xf0 [ 2581.161277] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2581.163821] ? kthread+0x1d1/0x200 [ 2581.165644] ? set_kthread_struct+0x70/0x70 [ 2581.167447] ? ret_from_fork+0x1f/0x30 [ 2596.060054] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 2596.062024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2596.063852] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [ 2596.066463] Modules linked in: lustre(O) osp(O) [ 2596.069519] mdt(O) mdd(O) mgs(O) lquota(O) [ 2596.072366] ofd(O) lod(O) [ 2596.085941] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2596.086031] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2596.086038] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2596.086044] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2596.086342] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2596.086353] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2596.086356] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2596.086361] RAX: 0000000000001b4b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2596.086363] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2596.086365] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2596.086367] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2596.086370] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2596.086373] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2596.086376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2596.086378] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2596.086455] Call Trace: [ 2596.086462] [ 2596.086466] ? show_regs.cold.9+0x22/0x2f [ 2596.086476] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2596.086482] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2596.086493] ? watchdog+0x40/0x40 [ 2596.086498] ? hrtimer_interrupt+0x128/0x2c0 [ 2596.086502] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2596.086510] ? apic_timer_interrupt+0xf/0x20 [ 2596.086515] [ 2596.086518] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2596.086524] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2596.086529] do_raw_spin_lock+0x177/0x190 [ 2596.086534] _raw_spin_lock+0x21/0x30 [ 2596.086539] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2596.087070] process_one_work+0x2c8/0x700 [ 2596.087077] worker_thread+0x296/0x6e0 [ 2596.087081] ? rescuer_thread+0x570/0x570 [ 2596.087085] kthread+0x1d1/0x200 [ 2596.087088] ? set_kthread_struct+0x70/0x70 [ 2596.087091] ret_from_fork+0x1f/0x30 [ 2596.233702] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2596.258656] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2596.263961] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2596.268530] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2596.271435] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2596.278488] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2596.286394] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2596.289890] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2596.295786] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2596.300472] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2596.303851] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2596.308894] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2596.313588] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2596.316839] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2596.322276] Call Trace: [ 2596.323691] [ 2596.324664] ? show_regs.cold.9+0x22/0x2f [ 2596.329501] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2596.331880] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2596.335468] ? watchdog+0x40/0x40 [ 2596.340893] ? hrtimer_interrupt+0x128/0x2c0 [ 2596.344589] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2596.354369] ? apic_timer_interrupt+0xf/0x20 [ 2596.358076] [ 2596.360646] ? native_safe_halt+0x2e/0x50 [ 2596.362859] kvm_wait+0xcd/0xe0 [ 2596.363863] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2596.372567] do_raw_spin_lock+0x177/0x190 [ 2596.383100] _raw_spin_lock+0x21/0x30 [ 2596.384240] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2596.392375] ? do_raw_spin_unlock+0x75/0x190 [ 2596.397563] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2596.406901] ? do_wait_intr+0xf0/0xf0 [ 2596.414481] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2596.426613] mdc_reint+0x2d/0x110 [mdc] [ 2596.431307] mdc_unlink+0x658/0x870 [mdc] [ 2596.440458] lmv_unlink+0x34d/0xbd0 [lmv] [ 2596.451389] ll_unlink+0x24d/0x7f0 [lustre] [ 2596.463409] vfs_unlink+0x202/0x330 [ 2596.471910] do_unlinkat+0x2fa/0x460 [ 2596.476726] __x64_sys_unlink+0x37/0x50 [ 2596.483300] do_syscall_64+0xc1/0x440 [ 2596.489581] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2596.502337] RIP: 0033:0x7f83e5f5c10b [ 2596.505073] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2596.530288] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2596.536099] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2596.542600] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2596.550919] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2596.558539] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2596.568260] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2606.051783] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 92s! [ 2606.069517] Showing busy workqueues and worker pools: [ 2606.082039] workqueue events: flags=0x0 [ 2606.092190] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2606.104153] pending: osc_grant_work_handler [osc], push_to_pool [ 2606.124333] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2606.133956] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 2606.139455] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2606.146840] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 2606.156745] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2606.163015] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 2606.172815] workqueue events_long: flags=0x0 [ 2606.178696] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2606.186154] pending: lu_objects_flush [obdclass] [ 2606.192534] workqueue events_unbound: flags=0x2 [ 2606.199224] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2606.201616] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2606.217985] workqueue events_power_efficient: flags=0x80 [ 2606.220787] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2606.223649] pending: neigh_periodic_work, do_cache_clean [ 2606.228370] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2606.232740] pending: neigh_periodic_work, check_lifetime [ 2606.237293] workqueue mm_percpu_wq: flags=0x8 [ 2606.240505] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2606.243763] pending: vmstat_update, lru_add_drain_per_cpu [ 2606.247531] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2606.255047] pending: vmstat_update [ 2606.260062] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2606.264567] pending: lru_add_drain_per_cpu BAR(48) [ 2606.266385] workqueue ipv6_addrconf: flags=0x40008 [ 2606.279415] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2606.286801] pending: addrconf_verify_work [ 2606.290616] workqueue knbd0-recv: flags=0x1a [ 2606.293225] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2606.297312] in-flight: 650:recv_work [ 2606.299840] workqueue ptlrpc_pinger: flags=0x2 [ 2606.307367] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2606.309632] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2606.321028] pool 8: cpus=0-3 flags=0x4 nice=0 hung=93s workers=4 idle: 36 466 29757 [ 2606.324244] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2596s workers=2 idle: 652 [ 2608.064840] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 2608.077471] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2608.115733] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2608.121300] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2608.125822] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2608.128984] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2608.139312] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2608.142853] RAX: 0000000000007311 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2608.147997] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 2608.151639] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2608.155344] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 2608.159956] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 2608.163644] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 2608.168470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2608.171223] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 2608.182483] Call Trace: [ 2608.183670] [ 2608.185116] ? show_regs.cold.9+0x22/0x2f [ 2608.190891] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2608.193422] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2608.198593] ? watchdog+0x40/0x40 [ 2608.200691] ? hrtimer_interrupt+0x128/0x2c0 [ 2608.205407] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2608.210542] ? apic_timer_interrupt+0xf/0x20 [ 2608.213075] [ 2608.214388] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2608.218044] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2608.221874] do_raw_spin_lock+0x177/0x190 [ 2608.223584] _raw_spin_lock+0x21/0x30 [ 2608.227083] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2608.230553] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2608.233456] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2608.239336] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2608.243327] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2608.247458] ? woken_wake_function+0x30/0x30 [ 2608.249533] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2608.252746] kthread+0x1d1/0x200 [ 2608.254088] ? set_kthread_struct+0x70/0x70 [ 2608.255875] ret_from_fork+0x1f/0x30 [ 2624.061150] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 2624.062057] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2624.066752] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) [ 2624.070818] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2624.070906] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2624.070914] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2624.070917] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2624.070930] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2624.070934] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2624.070939] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2624.070941] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2624.070943] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2624.070945] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2624.070947] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2624.070950] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2624.070954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2624.070956] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2624.070964] Call Trace: [ 2624.070970] [ 2624.070976] ? show_regs.cold.9+0x22/0x2f [ 2624.070986] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2624.070992] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2624.070996] ? watchdog+0x40/0x40 [ 2624.071001] ? hrtimer_interrupt+0x128/0x2c0 [ 2624.071005] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2624.071011] ? apic_timer_interrupt+0xf/0x20 [ 2624.071016] [ 2624.071019] ? native_safe_halt+0x2e/0x50 [ 2624.071023] kvm_wait+0xcd/0xe0 [ 2624.071030] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2624.071037] do_raw_spin_lock+0x177/0x190 [ 2624.071042] _raw_spin_lock+0x21/0x30 [ 2624.071047] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2624.071332] ? do_raw_spin_unlock+0x75/0x190 [ 2624.071339] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2624.081109] lnet(O) [ 2624.117083] ? do_wait_intr+0xf0/0xf0 [ 2624.129207] libcfs(O) [ 2624.133380] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2624.138174] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2624.138217] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2624.138224] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2624.138230] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2624.138646] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2624.138657] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2624.138661] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2624.138665] RAX: 0000000000006b90 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2624.138668] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2624.138670] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2624.138672] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2624.138674] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2624.138677] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2624.138679] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2624.138802] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2624.138810] Call Trace: [ 2624.138815] [ 2624.138820] ? show_regs.cold.9+0x22/0x2f [ 2624.138829] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2624.138834] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2624.138836] ? watchdog+0x40/0x40 [ 2624.138839] ? hrtimer_interrupt+0x128/0x2c0 [ 2624.138841] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2624.138847] ? apic_timer_interrupt+0xf/0x20 [ 2624.138851] [ 2624.138852] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2624.138855] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2624.138859] do_raw_spin_lock+0x177/0x190 [ 2624.138862] _raw_spin_lock+0x21/0x30 [ 2624.138865] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2624.139043] process_one_work+0x2c8/0x700 [ 2624.139048] worker_thread+0x296/0x6e0 [ 2624.139051] ? rescuer_thread+0x570/0x570 [ 2624.139053] kthread+0x1d1/0x200 [ 2624.139056] ? set_kthread_struct+0x70/0x70 [ 2624.139059] ret_from_fork+0x1f/0x30 [ 2624.405932] mdc_reint+0x2d/0x110 [mdc] [ 2624.407366] mdc_unlink+0x658/0x870 [mdc] [ 2624.409438] lmv_unlink+0x34d/0xbd0 [lmv] [ 2624.411102] ll_unlink+0x24d/0x7f0 [lustre] [ 2624.413888] vfs_unlink+0x202/0x330 [ 2624.416692] do_unlinkat+0x2fa/0x460 [ 2624.418440] __x64_sys_unlink+0x37/0x50 [ 2624.421201] do_syscall_64+0xc1/0x440 [ 2624.422553] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2624.428448] RIP: 0033:0x7f83e5f5c10b [ 2624.429784] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2624.443014] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2624.446201] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2624.451165] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2624.456053] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2624.459924] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2624.464105] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2632.056026] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 2632.066182] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2632.091998] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2632.097805] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2632.102480] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2632.105761] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2632.116818] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2632.121004] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2632.124972] RDX: 0000000000004340 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2632.128966] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2632.133667] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2632.137905] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2632.142006] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2632.146495] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2632.149681] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2632.153582] Call Trace: [ 2632.154829] [ 2632.155969] ? show_regs.cold.9+0x22/0x2f [ 2632.158197] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2632.161493] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2632.163928] ? watchdog+0x40/0x40 [ 2632.165786] ? hrtimer_interrupt+0x128/0x2c0 [ 2632.168750] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2632.171327] ? apic_timer_interrupt+0xf/0x20 [ 2632.173591] [ 2632.174959] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2632.178644] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2632.182000] do_raw_spin_lock+0x177/0x190 [ 2632.184962] _raw_spin_lock+0x21/0x30 [ 2632.187027] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2632.190498] obd_mod_free+0x46/0x140 [ptlrpc] [ 2632.193856] mdc_commit_open+0x91/0xb0 [mdc] [ 2632.196064] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2632.199822] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2632.203579] after_reply+0xe2c/0x1780 [ptlrpc] [ 2632.206105] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2632.209595] ? __next_timer_interrupt+0x160/0x160 [ 2632.212628] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2632.215044] ? do_wait_intr+0xf0/0xf0 [ 2632.217022] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2632.220544] kthread+0x1d1/0x200 [ 2632.222378] ? set_kthread_struct+0x70/0x70 [ 2632.224663] ret_from_fork+0x1f/0x30 [ 2636.059020] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 2636.064663] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2636.090350] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2636.102273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2636.107694] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2636.110834] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2636.120083] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2636.123688] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2636.126961] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 2636.130350] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2636.135956] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 2636.139986] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 2636.143483] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 2636.147834] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2636.151174] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 2636.154774] Call Trace: [ 2636.156020] [ 2636.157020] ? show_regs.cold.9+0x22/0x2f [ 2636.158969] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2636.161952] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2636.164998] ? watchdog+0x40/0x40 [ 2636.166698] ? hrtimer_interrupt+0x128/0x2c0 [ 2636.169035] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2636.171568] ? apic_timer_interrupt+0xf/0x20 [ 2636.174199] [ 2636.175308] ? native_safe_halt+0x2e/0x50 [ 2636.177442] kvm_wait+0xcd/0xe0 [ 2636.179053] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2636.181917] do_raw_spin_lock+0x177/0x190 [ 2636.183922] _raw_spin_lock+0x21/0x30 [ 2636.185807] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2636.188737] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2636.191128] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2636.193761] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2636.196273] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2636.198926] ? woken_wake_function+0x30/0x30 [ 2636.201229] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2636.204453] kthread+0x1d1/0x200 [ 2636.205557] ? set_kthread_struct+0x70/0x70 [ 2636.207227] ret_from_fork+0x1f/0x30 [ 2636.769160] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 123s! [ 2636.772946] Showing busy workqueues and worker pools: [ 2636.775357] workqueue events: flags=0x0 [ 2636.777353] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2636.780752] pending: osc_grant_work_handler [osc], push_to_pool [ 2636.783255] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2636.787446] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 2636.790717] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2636.794208] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 2636.798323] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2636.801471] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 2636.806030] workqueue events_long: flags=0x0 [ 2636.808254] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2636.812230] pending: lu_objects_flush [obdclass] [ 2636.814960] workqueue events_unbound: flags=0x2 [ 2636.817336] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2636.820890] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2636.824532] workqueue events_power_efficient: flags=0x80 [ 2636.827396] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2636.830960] pending: neigh_periodic_work, do_cache_clean [ 2636.833652] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2636.837267] pending: neigh_periodic_work, check_lifetime [ 2636.839953] workqueue mm_percpu_wq: flags=0x8 [ 2636.842409] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2636.845464] pending: vmstat_update, lru_add_drain_per_cpu [ 2636.848441] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2636.852211] pending: vmstat_update [ 2636.854051] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2636.857260] pending: lru_add_drain_per_cpu BAR(48) [ 2636.859948] workqueue ipv6_addrconf: flags=0x40008 [ 2636.862472] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2636.865430] pending: addrconf_verify_work [ 2636.867865] workqueue knbd0-recv: flags=0x1a [ 2636.870250] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2636.873512] in-flight: 650:recv_work [ 2636.875503] workqueue ptlrpc_pinger: flags=0x2 [ 2636.877775] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2636.880410] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2636.883473] pool 8: cpus=0-3 flags=0x4 nice=0 hung=123s workers=4 idle: 36 466 29757 [ 2636.887354] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2626s workers=2 idle: 652 [ 2652.060021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 2652.062471] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2652.065326] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2652.065404] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2652.065410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2652.065416] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2652.069172] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2652.069259] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2652.069267] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2652.069270] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2652.069286] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2652.069290] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2652.069375] RAX: 0000000000007f8a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2652.069378] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2652.069380] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2652.069382] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2652.069386] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2652.069389] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2652.069392] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2652.069395] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2652.069405] Call Trace: [ 2652.069411] [ 2652.069417] ? show_regs.cold.9+0x22/0x2f [ 2652.069445] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2652.069454] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2652.069460] ? watchdog+0x40/0x40 [ 2652.069466] ? hrtimer_interrupt+0x128/0x2c0 [ 2652.069470] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2652.069484] ? apic_timer_interrupt+0xf/0x20 [ 2652.069489] [ 2652.069492] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2652.069500] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2652.069506] do_raw_spin_lock+0x177/0x190 [ 2652.069512] _raw_spin_lock+0x21/0x30 [ 2652.069520] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2652.069997] ? do_raw_spin_unlock+0x75/0x190 [ 2652.070004] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2652.070215] ? do_wait_intr+0xf0/0xf0 [ 2652.070223] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2652.070521] mdc_reint+0x2d/0x110 [mdc] [ 2652.070622] mdc_unlink+0x658/0x870 [mdc] [ 2652.071000] lmv_unlink+0x34d/0xbd0 [lmv] [ 2652.071036] ll_unlink+0x24d/0x7f0 [lustre] [ 2652.071153] vfs_unlink+0x202/0x330 [ 2652.071162] do_unlinkat+0x2fa/0x460 [ 2652.071167] __x64_sys_unlink+0x37/0x50 [ 2652.071170] do_syscall_64+0xc1/0x440 [ 2652.071177] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2652.071183] RIP: 0033:0x7f83e5f5c10b [ 2652.071190] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2652.071194] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2652.071199] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2652.071201] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2652.071203] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2652.071205] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2652.071208] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2652.211892] [ 2652.212318] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2652.214157] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2652.220597] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2652.222829] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2652.225021] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2652.226948] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2652.228644] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2652.231038] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2652.233752] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2652.236706] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2652.239104] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2652.242672] Call Trace: [ 2652.243592] [ 2652.244341] ? show_regs.cold.9+0x22/0x2f [ 2652.245844] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2652.247772] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2652.249299] ? watchdog+0x40/0x40 [ 2652.250541] ? hrtimer_interrupt+0x128/0x2c0 [ 2652.251916] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2652.253606] ? apic_timer_interrupt+0xf/0x20 [ 2652.254881] [ 2652.255759] ? native_safe_halt+0x2e/0x50 [ 2652.257207] kvm_wait+0xcd/0xe0 [ 2652.263829] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2652.266451] do_raw_spin_lock+0x177/0x190 [ 2652.268395] _raw_spin_lock+0x21/0x30 [ 2652.270406] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2652.272835] process_one_work+0x2c8/0x700 [ 2652.274389] worker_thread+0x296/0x6e0 [ 2652.275783] ? rescuer_thread+0x570/0x570 [ 2652.277411] kthread+0x1d1/0x200 [ 2652.278697] ? set_kthread_struct+0x70/0x70 [ 2652.280189] ret_from_fork+0x1f/0x30 [ 2664.056024] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 2664.059026] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 2664.059947] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2664.060018] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2664.060026] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2664.060029] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2664.060186] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2664.060190] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2664.060195] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2664.060198] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2664.060200] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2664.060202] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2664.060205] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2664.060208] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2664.060211] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2664.060213] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2664.060222] Call Trace: [ 2664.060228] [ 2664.060233] ? show_regs.cold.9+0x22/0x2f [ 2664.060242] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2664.060248] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2664.060253] ? watchdog+0x40/0x40 [ 2664.060257] ? hrtimer_interrupt+0x128/0x2c0 [ 2664.060260] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2664.060268] ? apic_timer_interrupt+0xf/0x20 [ 2664.060273] [ 2664.060276] ? native_safe_halt+0x2e/0x50 [ 2664.060280] kvm_wait+0xcd/0xe0 [ 2664.060287] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2664.060293] do_raw_spin_lock+0x177/0x190 [ 2664.060297] _raw_spin_lock+0x21/0x30 [ 2664.060303] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2664.060959] obd_mod_free+0x46/0x140 [ptlrpc] [ 2664.063306] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2664.063359] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2664.063363] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2664.063366] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2664.063377] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2664.083482] mdc_commit_open+0x91/0xb0 [mdc] [ 2664.088356] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 [ 2664.092032] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2664.094527] ORIG_RAX: ffffffffffffff13 [ 2664.101958] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2664.102257] after_reply+0xe2c/0x1780 [ptlrpc] [ 2664.102462] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2664.102663] ? __next_timer_interrupt+0x160/0x160 [ 2664.102673] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2664.102946] ? do_wait_intr+0xf0/0xf0 [ 2664.102956] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2664.103164] kthread+0x1d1/0x200 [ 2664.103170] ? set_kthread_struct+0x70/0x70 [ 2664.103174] ret_from_fork+0x1f/0x30 [ 2664.249340] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2664.254016] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 2664.257431] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2664.261909] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 2664.265745] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 2664.270394] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 2664.274089] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2664.276767] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 2664.279421] Call Trace: [ 2664.281161] [ 2664.281908] ? show_regs.cold.9+0x22/0x2f [ 2664.283851] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2664.285611] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2664.290587] ? watchdog+0x40/0x40 [ 2664.292215] ? hrtimer_interrupt+0x128/0x2c0 [ 2664.296361] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2664.298731] ? apic_timer_interrupt+0xf/0x20 [ 2664.300761] [ 2664.301595] ? native_safe_halt+0x2e/0x50 [ 2664.303706] kvm_wait+0xcd/0xe0 [ 2664.304941] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2664.307486] do_raw_spin_lock+0x177/0x190 [ 2664.309898] _raw_spin_lock+0x21/0x30 [ 2664.311413] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2664.314588] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2664.320256] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2664.327558] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2664.335956] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2664.345656] ? woken_wake_function+0x30/0x30 [ 2664.352206] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2664.359726] kthread+0x1d1/0x200 [ 2664.365254] ? set_kthread_struct+0x70/0x70 [ 2664.369610] ret_from_fork+0x1f/0x30 [ 2666.982508] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 153s! [ 2666.995581] Showing busy workqueues and worker pools: [ 2667.003090] workqueue events: flags=0x0 [ 2667.005868] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2667.017569] pending: osc_grant_work_handler [osc], push_to_pool [ 2667.025475] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2667.036543] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 2667.048448] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2667.062109] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 2667.071779] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2667.081265] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 2667.088783] workqueue events_long: flags=0x0 [ 2667.099114] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2667.106633] pending: lu_objects_flush [obdclass] [ 2667.119148] workqueue events_unbound: flags=0x2 [ 2667.125472] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2667.134766] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2667.143413] workqueue events_power_efficient: flags=0x80 [ 2667.146669] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2667.158376] pending: neigh_periodic_work, do_cache_clean [ 2667.166170] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2667.171783] pending: neigh_periodic_work, check_lifetime [ 2667.174755] workqueue mm_percpu_wq: flags=0x8 [ 2667.176769] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2667.180745] pending: vmstat_update, lru_add_drain_per_cpu [ 2667.188867] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2667.191196] pending: vmstat_update [ 2667.203467] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2667.209346] pending: lru_add_drain_per_cpu BAR(48) [ 2667.212253] workqueue ipv6_addrconf: flags=0x40008 [ 2667.217641] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2667.224512] pending: addrconf_verify_work [ 2667.227627] workqueue knbd0-recv: flags=0x1a [ 2667.230189] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2667.237479] in-flight: 650:recv_work [ 2667.240865] workqueue ptlrpc_pinger: flags=0x2 [ 2667.245803] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2667.252220] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2667.255583] pool 8: cpus=0-3 flags=0x4 nice=0 hung=154s workers=4 idle: 36 466 29757 [ 2667.259551] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2657s workers=2 idle: 652 [ 2680.060027] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 2680.064263] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2680.066372] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 2680.092845] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2680.092858] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2680.092864] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2680.093225] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2680.093237] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2680.093242] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2680.093247] RAX: 0000000000004a0b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2680.093248] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2680.093251] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2680.093253] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2680.093255] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2680.093258] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2680.093261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2680.093263] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2680.093273] Call Trace: [ 2680.093278] [ 2680.093283] ? show_regs.cold.9+0x22/0x2f [ 2680.093293] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2680.093299] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2680.093303] ? watchdog+0x40/0x40 [ 2680.093308] ? hrtimer_interrupt+0x128/0x2c0 [ 2680.093312] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2680.093327] ? apic_timer_interrupt+0xf/0x20 [ 2680.093341] [ 2680.093343] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2680.093348] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2680.093353] do_raw_spin_lock+0x177/0x190 [ 2680.093358] _raw_spin_lock+0x21/0x30 [ 2680.093371] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2680.093675] process_one_work+0x2c8/0x700 [ 2680.093682] worker_thread+0x296/0x6e0 [ 2680.093686] ? rescuer_thread+0x570/0x570 [ 2680.093690] kthread+0x1d1/0x200 [ 2680.093694] ? set_kthread_struct+0x70/0x70 [ 2680.093697] ret_from_fork+0x1f/0x30 [ 2680.340974] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2680.385440] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2680.392226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2680.406506] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2680.412495] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2680.439763] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2680.446902] RAX: 0000000000000a48 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2680.451694] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2680.457661] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2680.466518] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2680.472016] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2680.479734] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2680.489633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2680.491664] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2680.505041] Call Trace: [ 2680.508932] [ 2680.513132] ? show_regs.cold.9+0x22/0x2f [ 2680.518002] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2680.522622] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2680.523764] ? watchdog+0x40/0x40 [ 2680.527754] ? hrtimer_interrupt+0x128/0x2c0 [ 2680.532913] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2680.538764] ? apic_timer_interrupt+0xf/0x20 [ 2680.543436] [ 2680.544282] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2680.552908] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2680.561583] do_raw_spin_lock+0x177/0x190 [ 2680.564970] _raw_spin_lock+0x21/0x30 [ 2680.567600] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2680.573032] ? do_raw_spin_unlock+0x75/0x190 [ 2680.579290] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2680.581109] ? do_wait_intr+0xf0/0xf0 [ 2680.582813] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2680.589020] mdc_reint+0x2d/0x110 [mdc] [ 2680.591163] mdc_unlink+0x658/0x870 [mdc] [ 2680.597484] lmv_unlink+0x34d/0xbd0 [lmv] [ 2680.600777] ll_unlink+0x24d/0x7f0 [lustre] [ 2680.605419] vfs_unlink+0x202/0x330 [ 2680.607589] do_unlinkat+0x2fa/0x460 [ 2680.610539] __x64_sys_unlink+0x37/0x50 [ 2680.613879] do_syscall_64+0xc1/0x440 [ 2680.620917] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2680.626018] RIP: 0033:0x7f83e5f5c10b [ 2680.628546] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2680.641424] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2680.653575] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2680.656754] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2680.666288] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2680.672752] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2680.680533] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2692.056118] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 2692.061402] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2692.062738] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 2692.089770] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2692.089786] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2692.089788] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2692.089810] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2692.089814] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2692.089820] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2692.089823] RDX: 000000000000357f RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2692.089825] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2692.089827] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2692.090017] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2692.090021] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2692.090024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2692.090026] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2692.090035] Call Trace: [ 2692.090053] [ 2692.090064] ? show_regs.cold.9+0x22/0x2f [ 2692.090074] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2692.090082] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2692.090086] ? watchdog+0x40/0x40 [ 2692.090090] ? hrtimer_interrupt+0x128/0x2c0 [ 2692.090094] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2692.090101] ? apic_timer_interrupt+0xf/0x20 [ 2692.090105] [ 2692.090108] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2692.090112] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2692.090116] do_raw_spin_lock+0x177/0x190 [ 2692.090120] _raw_spin_lock+0x21/0x30 [ 2692.090127] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2692.090393] obd_mod_free+0x46/0x140 [ptlrpc] [ 2692.090583] mdc_commit_open+0x91/0xb0 [mdc] [ 2692.090619] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2692.090825] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2692.091209] after_reply+0xe2c/0x1780 [ptlrpc] [ 2692.091411] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2692.091609] ? __next_timer_interrupt+0x160/0x160 [ 2692.091617] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2692.092208] ? do_wait_intr+0xf0/0xf0 [ 2692.092217] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2692.092594] kthread+0x1d1/0x200 [ 2692.092602] ? set_kthread_struct+0x70/0x70 [ 2692.092607] ret_from_fork+0x1f/0x30 [ 2692.213371] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2692.237615] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2692.243658] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2692.248181] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2692.251432] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2692.262601] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2692.267756] RAX: 0000000000003b09 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2692.271458] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 2692.275619] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2692.279252] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 2692.282715] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 2692.286774] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 2692.290379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2692.292628] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 2692.295606] Call Trace: [ 2692.297094] [ 2692.297762] ? show_regs.cold.9+0x22/0x2f [ 2692.299558] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2692.301337] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2692.303438] ? watchdog+0x40/0x40 [ 2692.304677] ? hrtimer_interrupt+0x128/0x2c0 [ 2692.307310] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2692.309572] ? apic_timer_interrupt+0xf/0x20 [ 2692.311739] [ 2692.312553] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2692.314416] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2692.316545] do_raw_spin_lock+0x177/0x190 [ 2692.318696] _raw_spin_lock+0x21/0x30 [ 2692.320291] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2692.323774] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2692.326312] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2692.329216] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2692.331376] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2692.334167] ? woken_wake_function+0x30/0x30 [ 2692.335823] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2692.338603] kthread+0x1d1/0x200 [ 2692.340083] ? set_kthread_struct+0x70/0x70 [ 2692.341683] ret_from_fork+0x1f/0x30 [ 2697.696081] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 184s! [ 2697.700087] Showing busy workqueues and worker pools: [ 2697.702855] workqueue events: flags=0x0 [ 2697.706167] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2697.710291] pending: osc_grant_work_handler [osc], push_to_pool [ 2697.714605] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2697.719472] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 2697.724608] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2697.731760] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 2697.736973] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2697.741411] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 2697.751878] workqueue events_long: flags=0x0 [ 2697.754218] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2697.758492] pending: lu_objects_flush [obdclass] [ 2697.762063] workqueue events_unbound: flags=0x2 [ 2697.767721] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2697.771488] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2697.783449] workqueue events_power_efficient: flags=0x80 [ 2697.786361] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2697.791394] pending: neigh_periodic_work, do_cache_clean [ 2697.796271] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2697.804387] pending: neigh_periodic_work, check_lifetime [ 2697.811789] workqueue mm_percpu_wq: flags=0x8 [ 2697.818328] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2697.827264] pending: vmstat_update, lru_add_drain_per_cpu [ 2697.832291] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2697.845103] pending: vmstat_update [ 2697.853487] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2697.856021] pending: lru_add_drain_per_cpu BAR(48) [ 2697.864927] workqueue ipv6_addrconf: flags=0x40008 [ 2697.866778] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2697.870999] pending: addrconf_verify_work [ 2697.875215] workqueue knbd0-recv: flags=0x1a [ 2697.877429] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2697.886235] in-flight: 650:recv_work [ 2697.893106] workqueue ptlrpc_pinger: flags=0x2 [ 2697.901894] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2697.907280] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2697.911215] pool 8: cpus=0-3 flags=0x4 nice=0 hung=184s workers=4 idle: 36 466 29757 [ 2697.915321] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2687s workers=2 idle: 652 [ 2708.060023] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 2708.062050] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 2708.064918] Modules linked in: [ 2708.068806] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2708.068996] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2708.069003] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2708.069006] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2708.069020] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2708.069025] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2708.069100] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2708.069103] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2708.069109] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2708.069111] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2708.069113] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2708.069117] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2708.069120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2708.069122] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2708.069131] Call Trace: [ 2708.069136] [ 2708.069142] ? show_regs.cold.9+0x22/0x2f [ 2708.069211] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2708.069217] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2708.069222] ? watchdog+0x40/0x40 [ 2708.069226] ? hrtimer_interrupt+0x128/0x2c0 [ 2708.069230] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2708.069240] ? apic_timer_interrupt+0xf/0x20 [ 2708.069245] [ 2708.069248] ? native_safe_halt+0x2e/0x50 [ 2708.069252] kvm_wait+0xcd/0xe0 [ 2708.069258] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2708.069265] do_raw_spin_lock+0x177/0x190 [ 2708.069270] _raw_spin_lock+0x21/0x30 [ 2708.069274] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2708.069608] ? do_raw_spin_unlock+0x75/0x190 [ 2708.069614] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2708.069838] ? do_wait_intr+0xf0/0xf0 [ 2708.070008] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2708.070813] mdc_reint+0x2d/0x110 [mdc] [ 2708.070864] mdc_unlink+0x658/0x870 [mdc] [ 2708.071135] lmv_unlink+0x34d/0xbd0 [lmv] [ 2708.071171] ll_unlink+0x24d/0x7f0 [lustre] [ 2708.071298] vfs_unlink+0x202/0x330 [ 2708.074143] do_unlinkat+0x2fa/0x460 [ 2708.074149] __x64_sys_unlink+0x37/0x50 [ 2708.074152] do_syscall_64+0xc1/0x440 [ 2708.074358] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2708.074370] RIP: 0033:0x7f83e5f5c10b [ 2708.074387] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2708.074459] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2708.074466] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2708.074469] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2708.074471] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2708.074473] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2708.074476] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2708.393218] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2708.436299] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2708.443054] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2708.446746] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2708.451005] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2708.455817] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2708.471595] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2708.476043] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2708.479723] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2708.484321] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2708.492605] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2708.498108] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2708.505926] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2708.510107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2708.516411] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2708.526457] Call Trace: [ 2708.528709] [ 2708.529436] ? show_regs.cold.9+0x22/0x2f [ 2708.532604] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2708.534644] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2708.539656] ? watchdog+0x40/0x40 [ 2708.541540] ? hrtimer_interrupt+0x128/0x2c0 [ 2708.544517] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2708.546980] ? apic_timer_interrupt+0xf/0x20 [ 2708.549019] [ 2708.550176] ? native_safe_halt+0x2e/0x50 [ 2708.551418] kvm_wait+0xcd/0xe0 [ 2708.552495] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2708.556253] do_raw_spin_lock+0x177/0x190 [ 2708.566553] _raw_spin_lock+0x21/0x30 [ 2708.568549] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2708.571625] process_one_work+0x2c8/0x700 [ 2708.573448] worker_thread+0x296/0x6e0 [ 2708.574824] ? rescuer_thread+0x570/0x570 [ 2708.576365] kthread+0x1d1/0x200 [ 2708.577717] ? set_kthread_struct+0x70/0x70 [ 2708.579445] ret_from_fork+0x1f/0x30 [ 2720.059021] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 2720.063983] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2720.083010] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2720.088646] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2720.092385] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2720.095113] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2720.101762] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2720.105155] RAX: 0000000000003007 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2720.107741] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 2720.110264] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2720.112655] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 2720.115164] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 2720.117541] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 2720.120059] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2720.122019] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 2720.124185] Call Trace: [ 2720.124989] [ 2720.125773] ? show_regs.cold.9+0x22/0x2f [ 2720.127222] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2720.129089] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2720.130750] ? watchdog+0x40/0x40 [ 2720.132046] ? hrtimer_interrupt+0x128/0x2c0 [ 2720.133733] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2720.135586] ? apic_timer_interrupt+0xf/0x20 [ 2720.137497] [ 2720.138204] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2720.140148] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2720.142183] do_raw_spin_lock+0x177/0x190 [ 2720.143740] _raw_spin_lock+0x21/0x30 [ 2720.145180] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2720.147944] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2720.150466] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2720.153184] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2720.156196] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2720.159232] ? woken_wake_function+0x30/0x30 [ 2720.161372] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2720.164289] kthread+0x1d1/0x200 [ 2720.165949] ? set_kthread_struct+0x70/0x70 [ 2720.167718] ret_from_fork+0x1f/0x30 [ 2724.056310] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 2724.061064] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2724.107763] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2724.120920] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2724.130360] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2724.136882] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2724.158597] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2724.166348] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2724.171060] RDX: 00000000000025df RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2724.178081] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2724.186081] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2724.193204] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2724.198978] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2724.206459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2724.221512] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2724.233200] Call Trace: [ 2724.239992] [ 2724.242019] ? show_regs.cold.9+0x22/0x2f [ 2724.243476] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2724.248741] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2724.254668] ? watchdog+0x40/0x40 [ 2724.257948] ? hrtimer_interrupt+0x128/0x2c0 [ 2724.261534] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2724.265717] ? apic_timer_interrupt+0xf/0x20 [ 2724.267238] [ 2724.271563] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2724.275241] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2724.279082] do_raw_spin_lock+0x177/0x190 [ 2724.281143] _raw_spin_lock+0x21/0x30 [ 2724.283001] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2724.286620] obd_mod_free+0x46/0x140 [ptlrpc] [ 2724.297001] mdc_commit_open+0x91/0xb0 [mdc] [ 2724.303978] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2724.312426] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2724.324964] after_reply+0xe2c/0x1780 [ptlrpc] [ 2724.334137] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2724.338646] ? __next_timer_interrupt+0x160/0x160 [ 2724.343243] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2724.348681] ? do_wait_intr+0xf0/0xf0 [ 2724.351119] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2724.354434] kthread+0x1d1/0x200 [ 2724.356618] ? set_kthread_struct+0x70/0x70 [ 2724.359255] ret_from_fork+0x1f/0x30 [ 2728.417085] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 215s! [ 2728.420721] Showing busy workqueues and worker pools: [ 2728.423168] workqueue events: flags=0x0 [ 2728.425518] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2728.431583] pending: osc_grant_work_handler [osc], push_to_pool [ 2728.439870] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2728.444157] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 2728.449621] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2728.453893] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 2728.460282] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2728.471580] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 2728.486968] workqueue events_long: flags=0x0 [ 2728.496592] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2728.500127] pending: lu_objects_flush [obdclass] [ 2728.502560] workqueue events_unbound: flags=0x2 [ 2728.505209] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2728.510928] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2728.516337] workqueue events_power_efficient: flags=0x80 [ 2728.519336] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2728.522530] pending: neigh_periodic_work, do_cache_clean [ 2728.524367] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2728.528441] pending: neigh_periodic_work, check_lifetime [ 2728.535439] workqueue mm_percpu_wq: flags=0x8 [ 2728.537747] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2728.541289] pending: vmstat_update, lru_add_drain_per_cpu [ 2728.547449] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2728.550477] pending: vmstat_update [ 2728.553586] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2728.558797] pending: lru_add_drain_per_cpu BAR(48) [ 2728.561422] workqueue ipv6_addrconf: flags=0x40008 [ 2728.563213] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2728.567841] pending: addrconf_verify_work [ 2728.571962] workqueue knbd0-recv: flags=0x1a [ 2728.574984] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2728.579063] in-flight: 650:recv_work [ 2728.581706] workqueue ptlrpc_pinger: flags=0x2 [ 2728.583643] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2728.586745] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2728.593402] pool 8: cpus=0-3 flags=0x4 nice=0 hung=215s workers=4 idle: 36 466 29757 [ 2728.598081] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2718s workers=2 idle: 652 [ 2736.060830] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 2736.062036] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 2736.065844] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2736.065928] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2736.065935] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2736.065942] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2736.066220] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 2736.066231] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 2736.066234] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2736.066238] RAX: 0000000000006a09 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2736.066241] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2736.066242] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2736.066244] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2736.066246] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2736.066249] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2736.066251] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2736.066253] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2736.066262] Call Trace: [ 2736.066268] [ 2736.066274] ? show_regs.cold.9+0x22/0x2f [ 2736.066283] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2736.066289] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2736.066293] ? watchdog+0x40/0x40 [ 2736.066297] ? hrtimer_interrupt+0x128/0x2c0 [ 2736.066301] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2736.066309] ? apic_timer_interrupt+0xf/0x20 [ 2736.066313] [ 2736.066316] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 2736.066320] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2736.066325] do_raw_spin_lock+0x177/0x190 [ 2736.066329] _raw_spin_lock+0x21/0x30 [ 2736.066333] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2736.073375] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2736.073461] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2736.073469] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2736.073472] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2736.073486] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2736.073490] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2736.073495] RAX: 0000000000001684 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2736.073497] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2736.073499] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2736.073500] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2736.073502] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2736.073505] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2736.073508] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2736.073510] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2736.073517] Call Trace: [ 2736.073522] [ 2736.073528] ? show_regs.cold.9+0x22/0x2f [ 2736.073536] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2736.073543] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2736.073547] ? watchdog+0x40/0x40 [ 2736.073551] ? hrtimer_interrupt+0x128/0x2c0 [ 2736.073555] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2736.073564] ? apic_timer_interrupt+0xf/0x20 [ 2736.073569] [ 2736.073572] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2736.073577] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2736.073582] do_raw_spin_lock+0x177/0x190 [ 2736.073587] _raw_spin_lock+0x21/0x30 [ 2736.073592] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2736.073867] ? do_raw_spin_unlock+0x75/0x190 [ 2736.073873] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2736.074582] ? do_wait_intr+0xf0/0xf0 [ 2736.074591] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2736.074792] mdc_reint+0x2d/0x110 [mdc] [ 2736.074828] mdc_unlink+0x658/0x870 [mdc] [ 2736.074854] lmv_unlink+0x34d/0xbd0 [lmv] [ 2736.074884] ll_unlink+0x24d/0x7f0 [lustre] [ 2736.103706] process_one_work+0x2c8/0x700 [ 2736.103727] worker_thread+0x296/0x6e0 [ 2736.103734] ? rescuer_thread+0x570/0x570 [ 2736.103737] kthread+0x1d1/0x200 [ 2736.103741] ? set_kthread_struct+0x70/0x70 [ 2736.103745] ret_from_fork+0x1f/0x30 [ 2736.361386] vfs_unlink+0x202/0x330 [ 2736.363080] do_unlinkat+0x2fa/0x460 [ 2736.364787] __x64_sys_unlink+0x37/0x50 [ 2736.366777] do_syscall_64+0xc1/0x440 [ 2736.368313] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2736.370723] RIP: 0033:0x7f83e5f5c10b [ 2736.372209] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2736.380354] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2736.383770] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2736.386909] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2736.390787] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2736.394142] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2736.397208] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2748.059036] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 2748.062333] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2748.082418] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2748.089479] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2748.093800] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2748.097372] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2748.110449] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2748.119041] RAX: 000000000000072d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2748.124758] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 2748.129407] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2748.138019] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 2748.147006] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 2748.152049] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 2748.157439] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2748.164350] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 2748.168460] Call Trace: [ 2748.174214] [ 2748.174990] ? show_regs.cold.9+0x22/0x2f [ 2748.179819] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2748.184746] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2748.187975] ? watchdog+0x40/0x40 [ 2748.190833] ? hrtimer_interrupt+0x128/0x2c0 [ 2748.193864] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2748.198112] ? apic_timer_interrupt+0xf/0x20 [ 2748.200788] [ 2748.201669] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2748.204813] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2748.209874] do_raw_spin_lock+0x177/0x190 [ 2748.215181] _raw_spin_lock+0x21/0x30 [ 2748.217416] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2748.224399] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2748.232420] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2748.242593] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2748.250198] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2748.257646] ? woken_wake_function+0x30/0x30 [ 2748.261459] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2748.265905] kthread+0x1d1/0x200 [ 2748.268046] ? set_kthread_struct+0x70/0x70 [ 2748.271264] ret_from_fork+0x1f/0x30 [ 2756.061513] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 2756.068346] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2756.120059] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2756.128356] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2756.133759] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2756.137023] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2756.158786] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2756.165010] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2756.167908] RDX: 0000000000004b2c RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2756.178978] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2756.183940] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2756.189031] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2756.197066] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2756.203633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2756.207936] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2756.215859] Call Trace: [ 2756.222925] [ 2756.229328] ? show_regs.cold.9+0x22/0x2f [ 2756.230902] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2756.233737] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2756.235949] ? watchdog+0x40/0x40 [ 2756.237692] ? hrtimer_interrupt+0x128/0x2c0 [ 2756.239741] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2756.242312] ? apic_timer_interrupt+0xf/0x20 [ 2756.244693] [ 2756.245750] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2756.248305] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2756.251070] do_raw_spin_lock+0x177/0x190 [ 2756.253189] _raw_spin_lock+0x21/0x30 [ 2756.254974] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2756.261901] obd_mod_free+0x46/0x140 [ptlrpc] [ 2756.265055] mdc_commit_open+0x91/0xb0 [mdc] [ 2756.272990] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2756.283690] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2756.290664] after_reply+0xe2c/0x1780 [ptlrpc] [ 2756.294841] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2756.300442] ? __next_timer_interrupt+0x160/0x160 [ 2756.303182] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2756.305295] ? do_wait_intr+0xf0/0xf0 [ 2756.307968] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2756.310177] kthread+0x1d1/0x200 [ 2756.311604] ? set_kthread_struct+0x70/0x70 [ 2756.313109] ret_from_fork+0x1f/0x30 [ 2758.627811] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 245s! [ 2758.637531] Showing busy workqueues and worker pools: [ 2758.639469] workqueue events: flags=0x0 [ 2758.641069] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2758.648209] pending: osc_grant_work_handler [osc], push_to_pool [ 2758.657032] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2758.660925] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 2758.662996] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2758.666716] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 2758.672642] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2758.676524] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 2758.687897] workqueue events_long: flags=0x0 [ 2758.691260] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2758.695694] pending: lu_objects_flush [obdclass] [ 2758.699207] workqueue events_unbound: flags=0x2 [ 2758.702219] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2758.705927] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2758.709606] workqueue events_power_efficient: flags=0x80 [ 2758.712305] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2758.715754] pending: neigh_periodic_work, do_cache_clean [ 2758.719221] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2758.722554] pending: neigh_periodic_work, check_lifetime [ 2758.725658] workqueue mm_percpu_wq: flags=0x8 [ 2758.729375] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2758.733214] pending: vmstat_update, lru_add_drain_per_cpu [ 2758.736591] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2758.739926] pending: vmstat_update [ 2758.741955] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2758.746022] pending: lru_add_drain_per_cpu BAR(48) [ 2758.748947] workqueue ipv6_addrconf: flags=0x40008 [ 2758.750704] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2758.753797] pending: addrconf_verify_work [ 2758.756224] workqueue knbd0-recv: flags=0x1a [ 2758.758486] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2758.761828] in-flight: 650:recv_work [ 2758.764142] workqueue ptlrpc_pinger: flags=0x2 [ 2758.766813] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2758.769543] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2758.778452] pool 8: cpus=0-3 flags=0x4 nice=0 hung=245s workers=4 idle: 36 466 29757 [ 2758.787012] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2748s workers=2 idle: 652 [ 2760.973181] rcu: INFO: rcu_sched self-detected stall on CPU [ 2760.976067] rcu: 0-....: (517164 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 2760.980848] (t=780021 jiffies g=262441 q=57149) [ 2760.982373] rcu: rcu_sched kthread starved for 249210 jiffies! g262441 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 2760.988181] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2760.992142] rcu: RCU grace-period kthread stack dump: [ 2760.994402] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2760.998775] Call Trace: [ 2760.999840] __schedule+0x351/0xcb0 [ 2761.001310] schedule+0xc0/0x180 [ 2761.002537] schedule_timeout+0xb4/0x190 [ 2761.004094] ? do_raw_spin_unlock+0x75/0x190 [ 2761.006158] ? __next_timer_interrupt+0x160/0x160 [ 2761.008353] ? prepare_to_swait_event+0xf3/0x140 [ 2761.010637] rcu_gp_kthread+0x8dd/0x15b0 [ 2761.012111] ? force_qs_rnp+0x2c0/0x2c0 [ 2761.014764] kthread+0x1d1/0x200 [ 2761.016236] ? set_kthread_struct+0x70/0x70 [ 2761.018122] ret_from_fork+0x1f/0x30 [ 2761.019568] rcu: Stack dump where RCU GP kthread last ran: [ 2761.022133] NMI backtrace for cpu 0 [ 2761.023440] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2761.028195] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2761.032546] Call Trace: [ 2761.033344] [ 2761.034747] ? dump_stack+0xbb/0x10e [ 2761.036645] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2761.038909] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2761.041209] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2761.043824] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2761.046293] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2761.048898] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 2761.051633] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 2761.053837] ? update_process_times+0x6d/0xc0 [ 2761.055987] ? tick_sched_handle.isra.18+0x27/0x90 [ 2761.058612] ? tick_sched_timer+0x53/0xb0 [ 2761.060245] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2761.062870] ? tick_sched_handle.isra.18+0x90/0x90 [ 2761.065007] ? hrtimer_interrupt+0x128/0x2c0 [ 2761.067090] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2761.069924] ? apic_timer_interrupt+0xf/0x20 [ 2761.072233] [ 2761.074090] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2761.076976] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2761.080566] ? do_raw_spin_lock+0x177/0x190 [ 2761.082296] ? _raw_spin_lock+0x21/0x30 [ 2761.084220] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2761.087250] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2761.089896] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2761.092542] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2761.095327] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2761.098293] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2761.100754] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2761.103757] ? __next_timer_interrupt+0x160/0x160 [ 2761.106050] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2761.108563] ? do_wait_intr+0xf0/0xf0 [ 2761.109699] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2761.117236] ? kthread+0x1d1/0x200 [ 2761.118999] ? set_kthread_struct+0x70/0x70 [ 2761.123573] ? ret_from_fork+0x1f/0x30 [ 2761.125882] NMI backtrace for cpu 0 [ 2761.127534] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2761.131838] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2761.135816] Call Trace: [ 2761.136702] [ 2761.137658] ? dump_stack+0xbb/0x10e [ 2761.139218] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2761.141737] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2761.143845] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2761.145570] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2761.149319] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2761.152334] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2761.158805] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2761.160613] ? update_process_times+0x6d/0xc0 [ 2761.162304] ? tick_sched_handle.isra.18+0x27/0x90 [ 2761.163840] ? tick_sched_timer+0x53/0xb0 [ 2761.167917] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2761.171240] ? tick_sched_handle.isra.18+0x90/0x90 [ 2761.174599] ? hrtimer_interrupt+0x128/0x2c0 [ 2761.177029] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2761.178951] ? apic_timer_interrupt+0xf/0x20 [ 2761.183576] [ 2761.187364] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2761.194924] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2761.200560] ? do_raw_spin_lock+0x177/0x190 [ 2761.206504] ? _raw_spin_lock+0x21/0x30 [ 2761.208961] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2761.215777] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2761.219998] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2761.223065] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2761.228327] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2761.232509] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2761.237876] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2761.243001] ? __next_timer_interrupt+0x160/0x160 [ 2761.250386] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2761.256819] ? do_wait_intr+0xf0/0xf0 [ 2761.264696] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2761.269345] ? kthread+0x1d1/0x200 [ 2761.275840] ? set_kthread_struct+0x70/0x70 [ 2761.278560] ? ret_from_fork+0x1f/0x30 [ 2764.060026] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 2764.063474] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 2764.063648] Modules linked in: [ 2764.078195] Modules linked in: [ 2764.078198] lustre(O) osp(O) [ 2764.079288] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2764.079410] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2764.079417] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2764.079419] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2764.079432] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2764.079436] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2764.079440] RAX: 000000000000088e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2764.079443] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2764.079445] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2764.079447] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2764.079449] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2764.079453] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2764.079455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2764.079458] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2764.079465] Call Trace: [ 2764.079469] [ 2764.079473] ? show_regs.cold.9+0x22/0x2f [ 2764.079481] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2764.079488] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2764.079492] ? watchdog+0x40/0x40 [ 2764.079496] ? hrtimer_interrupt+0x128/0x2c0 [ 2764.079499] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2764.079507] ? apic_timer_interrupt+0xf/0x20 [ 2764.079511] [ 2764.079514] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2764.079518] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2764.079522] do_raw_spin_lock+0x177/0x190 [ 2764.079526] _raw_spin_lock+0x21/0x30 [ 2764.079532] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2764.079801] ? do_raw_spin_unlock+0x75/0x190 [ 2764.079806] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2764.081206] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2764.081274] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2764.081280] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2764.081286] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2764.081554] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2764.081565] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2764.081569] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2764.081573] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2764.081576] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2764.081578] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2764.081579] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2764.081582] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2764.081584] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2764.081587] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2764.081589] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2764.081598] Call Trace: [ 2764.081602] [ 2764.081607] ? show_regs.cold.9+0x22/0x2f [ 2764.081623] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2764.081629] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2764.081633] ? watchdog+0x40/0x40 [ 2764.081637] ? hrtimer_interrupt+0x128/0x2c0 [ 2764.081641] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2764.081649] ? apic_timer_interrupt+0xf/0x20 [ 2764.081654] [ 2764.081656] ? native_safe_halt+0x2e/0x50 [ 2764.081659] kvm_wait+0xcd/0xe0 [ 2764.081667] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2764.081674] do_raw_spin_lock+0x177/0x190 [ 2764.081678] _raw_spin_lock+0x21/0x30 [ 2764.081683] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2764.081873] process_one_work+0x2c8/0x700 [ 2764.081878] worker_thread+0x296/0x6e0 [ 2764.081881] ? rescuer_thread+0x570/0x570 [ 2764.081884] kthread+0x1d1/0x200 [ 2764.081887] ? set_kthread_struct+0x70/0x70 [ 2764.081889] ret_from_fork+0x1f/0x30 [ 2764.456895] ? do_wait_intr+0xf0/0xf0 [ 2764.460627] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2764.463377] mdc_reint+0x2d/0x110 [mdc] [ 2764.464583] mdc_unlink+0x658/0x870 [mdc] [ 2764.466876] lmv_unlink+0x34d/0xbd0 [lmv] [ 2764.468850] ll_unlink+0x24d/0x7f0 [lustre] [ 2764.470740] vfs_unlink+0x202/0x330 [ 2764.471920] do_unlinkat+0x2fa/0x460 [ 2764.474898] __x64_sys_unlink+0x37/0x50 [ 2764.476525] do_syscall_64+0xc1/0x440 [ 2764.477881] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2764.482382] RIP: 0033:0x7f83e5f5c10b [ 2764.484727] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2764.494622] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2764.498590] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2764.501698] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2764.506631] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2764.508869] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2764.512378] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2776.059032] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 2776.062188] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2776.102838] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2776.115751] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2776.123533] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2776.125312] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2776.151740] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2776.164425] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2776.166533] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 2776.179048] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2776.186222] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 2776.194303] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 2776.209542] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 2776.227707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2776.233550] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 2776.236760] Call Trace: [ 2776.237891] [ 2776.239003] ? show_regs.cold.9+0x22/0x2f [ 2776.246517] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2776.249790] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2776.253006] ? watchdog+0x40/0x40 [ 2776.255059] ? hrtimer_interrupt+0x128/0x2c0 [ 2776.257521] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2776.260265] ? apic_timer_interrupt+0xf/0x20 [ 2776.263523] [ 2776.264736] ? native_safe_halt+0x2e/0x50 [ 2776.266777] kvm_wait+0xcd/0xe0 [ 2776.269191] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2776.272034] do_raw_spin_lock+0x177/0x190 [ 2776.274444] _raw_spin_lock+0x21/0x30 [ 2776.276737] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2776.280294] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2776.283012] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2776.285973] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2776.288923] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2776.291965] ? woken_wake_function+0x30/0x30 [ 2776.293951] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2776.296779] kthread+0x1d1/0x200 [ 2776.298536] ? set_kthread_struct+0x70/0x70 [ 2776.299952] ret_from_fork+0x1f/0x30 [ 2788.056027] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 2788.066470] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2788.105778] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2788.111071] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2788.115095] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2788.117376] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2788.126808] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2788.130225] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2788.133519] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2788.136779] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2788.139919] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2788.142603] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2788.145082] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2788.148318] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2788.154780] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2788.158601] Call Trace: [ 2788.159757] [ 2788.160729] ? show_regs.cold.9+0x22/0x2f [ 2788.162587] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2788.164450] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2788.165981] ? watchdog+0x40/0x40 [ 2788.166978] ? hrtimer_interrupt+0x128/0x2c0 [ 2788.171208] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2788.173942] ? apic_timer_interrupt+0xf/0x20 [ 2788.176718] [ 2788.177829] ? native_safe_halt+0x2e/0x50 [ 2788.180074] kvm_wait+0xcd/0xe0 [ 2788.181738] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2788.184365] do_raw_spin_lock+0x177/0x190 [ 2788.186329] _raw_spin_lock+0x21/0x30 [ 2788.188033] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2788.190822] obd_mod_free+0x46/0x140 [ptlrpc] [ 2788.193301] mdc_commit_open+0x91/0xb0 [mdc] [ 2788.195491] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2788.198232] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2788.200827] after_reply+0xe2c/0x1780 [ptlrpc] [ 2788.202775] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2788.205742] ? __next_timer_interrupt+0x160/0x160 [ 2788.207930] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2788.210560] ? do_wait_intr+0xf0/0xf0 [ 2788.212342] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2788.215022] kthread+0x1d1/0x200 [ 2788.216767] ? set_kthread_struct+0x70/0x70 [ 2788.218778] ret_from_fork+0x1f/0x30 [ 2788.833975] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 275s! [ 2788.843665] Showing busy workqueues and worker pools: [ 2788.849063] workqueue events: flags=0x0 [ 2788.853513] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2788.856499] pending: osc_grant_work_handler [osc], push_to_pool [ 2788.861430] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2788.865627] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 2788.871683] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2788.881058] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 2788.889742] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2788.893631] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 2788.899444] workqueue events_long: flags=0x0 [ 2788.901760] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2788.905762] pending: lu_objects_flush [obdclass] [ 2788.909797] workqueue events_unbound: flags=0x2 [ 2788.915633] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2788.924815] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2788.934494] workqueue events_power_efficient: flags=0x80 [ 2788.940705] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2788.950081] pending: neigh_periodic_work, do_cache_clean [ 2788.959815] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2788.962812] pending: neigh_periodic_work, check_lifetime [ 2788.966093] workqueue mm_percpu_wq: flags=0x8 [ 2788.970656] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2788.975761] pending: vmstat_update, lru_add_drain_per_cpu [ 2788.982067] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2788.986610] pending: vmstat_update [ 2788.989503] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2788.996583] pending: lru_add_drain_per_cpu BAR(48) [ 2789.003877] workqueue ipv6_addrconf: flags=0x40008 [ 2789.011929] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2789.018480] pending: addrconf_verify_work [ 2789.022907] workqueue knbd0-recv: flags=0x1a [ 2789.026090] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2789.035586] in-flight: 650:recv_work [ 2789.042080] workqueue ptlrpc_pinger: flags=0x2 [ 2789.045417] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2789.050505] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2789.055494] pool 8: cpus=0-3 flags=0x4 nice=0 hung=275s workers=4 idle: 36 466 29757 [ 2789.061827] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2779s workers=2 idle: 652 [ 2792.060029] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 2792.069392] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) [ 2792.069441] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 2792.069451] Modules linked in: [ 2792.087464] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2792.087677] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2792.087685] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2792.087691] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2792.087995] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2792.088007] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2792.088011] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2792.088015] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2792.088018] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2792.088020] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2792.088023] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2792.088026] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2792.088030] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2792.088033] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2792.088035] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2792.088045] Call Trace: [ 2792.088051] [ 2792.088056] ? show_regs.cold.9+0x22/0x2f [ 2792.088254] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2792.088264] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2792.088269] ? watchdog+0x40/0x40 [ 2792.088273] ? hrtimer_interrupt+0x128/0x2c0 [ 2792.088277] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2792.088284] ? apic_timer_interrupt+0xf/0x20 [ 2792.088288] [ 2792.088291] ? native_safe_halt+0x2e/0x50 [ 2792.088295] kvm_wait+0xcd/0xe0 [ 2792.088303] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2792.088310] do_raw_spin_lock+0x177/0x190 [ 2792.088315] _raw_spin_lock+0x21/0x30 [ 2792.088319] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2792.089459] process_one_work+0x2c8/0x700 [ 2792.089469] worker_thread+0x296/0x6e0 [ 2792.089475] ? rescuer_thread+0x570/0x570 [ 2792.100719] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2792.100792] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2792.100800] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2792.100802] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2792.100815] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2792.100819] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2792.100823] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2792.100825] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2792.100827] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2792.100830] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2792.100832] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2792.100835] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2792.100838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2792.100840] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2792.100851] Call Trace: [ 2792.100856] [ 2792.100861] ? show_regs.cold.9+0x22/0x2f [ 2792.100881] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2792.100887] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2792.100895] ? watchdog+0x40/0x40 [ 2792.100899] ? hrtimer_interrupt+0x128/0x2c0 [ 2792.100904] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2792.100910] ? apic_timer_interrupt+0xf/0x20 [ 2792.100915] [ 2792.100918] ? native_safe_halt+0x2e/0x50 [ 2792.100922] kvm_wait+0xcd/0xe0 [ 2792.100928] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2792.100936] do_raw_spin_lock+0x177/0x190 [ 2792.100939] _raw_spin_lock+0x21/0x30 [ 2792.100945] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2792.101346] ? do_raw_spin_unlock+0x75/0x190 [ 2792.101352] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2792.101720] ? do_wait_intr+0xf0/0xf0 [ 2792.101729] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2792.103960] kthread+0x1d1/0x200 [ 2792.103976] ? set_kthread_struct+0x70/0x70 [ 2792.103981] ret_from_fork+0x1f/0x30 [ 2792.534676] mdc_reint+0x2d/0x110 [mdc] [ 2792.536452] mdc_unlink+0x658/0x870 [mdc] [ 2792.538804] lmv_unlink+0x34d/0xbd0 [lmv] [ 2792.542630] ll_unlink+0x24d/0x7f0 [lustre] [ 2792.545465] vfs_unlink+0x202/0x330 [ 2792.546825] do_unlinkat+0x2fa/0x460 [ 2792.551410] __x64_sys_unlink+0x37/0x50 [ 2792.553211] do_syscall_64+0xc1/0x440 [ 2792.554424] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2792.558391] RIP: 0033:0x7f83e5f5c10b [ 2792.559944] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2792.571467] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2792.574895] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2792.585561] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2792.590974] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2792.595976] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2792.602691] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2804.064414] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 2804.072602] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2804.112681] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2804.125202] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2804.129656] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2804.132334] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2804.145003] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2804.148514] RAX: 0000000000007c11 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2804.152939] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 2804.156943] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2804.160519] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 2804.163073] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 2804.166401] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 2804.169160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2804.171544] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 2804.177394] Call Trace: [ 2804.179407] [ 2804.180191] ? show_regs.cold.9+0x22/0x2f [ 2804.185515] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2804.191725] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2804.199146] ? watchdog+0x40/0x40 [ 2804.203359] ? hrtimer_interrupt+0x128/0x2c0 [ 2804.205745] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2804.212519] ? apic_timer_interrupt+0xf/0x20 [ 2804.221875] [ 2804.223594] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2804.228002] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2804.230711] do_raw_spin_lock+0x177/0x190 [ 2804.233413] _raw_spin_lock+0x21/0x30 [ 2804.235381] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2804.239813] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2804.243701] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2804.252459] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2804.258211] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2804.261089] ? woken_wake_function+0x30/0x30 [ 2804.263147] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2804.267306] kthread+0x1d1/0x200 [ 2804.269031] ? set_kthread_struct+0x70/0x70 [ 2804.270916] ret_from_fork+0x1f/0x30 [ 2816.056009] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 2816.057961] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2816.100236] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2816.116301] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2816.126806] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2816.136442] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2816.151790] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2816.156015] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2816.162496] RDX: 0000000000006465 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2816.166039] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2816.168541] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2816.177473] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2816.188287] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2816.199307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2816.201504] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2816.204697] Call Trace: [ 2816.206829] [ 2816.216011] ? show_regs.cold.9+0x22/0x2f [ 2816.223181] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2816.229198] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2816.231432] ? watchdog+0x40/0x40 [ 2816.233557] ? hrtimer_interrupt+0x128/0x2c0 [ 2816.234689] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2816.236659] ? apic_timer_interrupt+0xf/0x20 [ 2816.238331] [ 2816.239149] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2816.241263] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2816.243506] do_raw_spin_lock+0x177/0x190 [ 2816.244781] _raw_spin_lock+0x21/0x30 [ 2816.246382] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2816.248378] obd_mod_free+0x46/0x140 [ptlrpc] [ 2816.250600] mdc_commit_open+0x91/0xb0 [mdc] [ 2816.252332] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2816.254641] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2816.257471] after_reply+0xe2c/0x1780 [ptlrpc] [ 2816.259682] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2816.262057] ? __next_timer_interrupt+0x160/0x160 [ 2816.264199] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2816.266411] ? do_wait_intr+0xf0/0xf0 [ 2816.268115] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2816.270385] kthread+0x1d1/0x200 [ 2816.271660] ? set_kthread_struct+0x70/0x70 [ 2816.273300] ret_from_fork+0x1f/0x30 [ 2819.554026] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 306s! [ 2819.559013] Showing busy workqueues and worker pools: [ 2819.562396] workqueue events: flags=0x0 [ 2819.564203] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2819.567047] pending: osc_grant_work_handler [osc], push_to_pool [ 2819.569469] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2819.571957] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 2819.574533] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2819.577282] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 2819.580691] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2819.583322] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 2819.588091] workqueue events_long: flags=0x0 [ 2819.589427] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2819.591690] pending: lu_objects_flush [obdclass] [ 2819.593853] workqueue events_unbound: flags=0x2 [ 2819.595380] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2819.597157] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2819.599091] workqueue events_power_efficient: flags=0x80 [ 2819.600663] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2819.602627] pending: neigh_periodic_work, do_cache_clean [ 2819.604163] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2819.606144] pending: neigh_periodic_work, check_lifetime [ 2819.607927] workqueue mm_percpu_wq: flags=0x8 [ 2819.609133] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2819.611067] pending: vmstat_update, lru_add_drain_per_cpu [ 2819.612719] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2819.614639] pending: vmstat_update [ 2819.615877] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2819.617684] pending: lru_add_drain_per_cpu BAR(48) [ 2819.619141] workqueue ipv6_addrconf: flags=0x40008 [ 2819.620378] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2819.622420] pending: addrconf_verify_work [ 2819.623703] workqueue knbd0-recv: flags=0x1a [ 2819.625013] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2819.626928] in-flight: 650:recv_work [ 2819.628110] workqueue ptlrpc_pinger: flags=0x2 [ 2819.629360] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2819.631223] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2819.633163] pool 8: cpus=0-3 flags=0x4 nice=0 hung=306s workers=4 idle: 36 466 29757 [ 2819.635387] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2809s workers=2 idle: 652 [ 2820.060363] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 2820.064529] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2820.086757] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2820.092018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2820.095917] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2820.099553] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2820.102900] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2820.111159] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2820.115244] RAX: 0000000000002616 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2820.118558] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2820.121715] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2820.125097] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2820.128522] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2820.132558] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2820.138503] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2820.143464] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2820.147730] Call Trace: [ 2820.149686] [ 2820.151478] ? show_regs.cold.9+0x22/0x2f [ 2820.153048] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2820.156123] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2820.158593] ? watchdog+0x40/0x40 [ 2820.161110] ? hrtimer_interrupt+0x128/0x2c0 [ 2820.163969] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2820.167057] ? apic_timer_interrupt+0xf/0x20 [ 2820.168293] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2820.169478] [ 2820.169490] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2820.169502] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2820.169509] do_raw_spin_lock+0x177/0x190 [ 2820.169515] _raw_spin_lock+0x21/0x30 [ 2820.169522] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2820.169869] process_one_work+0x2c8/0x700 [ 2820.169877] worker_thread+0x296/0x6e0 [ 2820.169886] ? rescuer_thread+0x570/0x570 [ 2820.169890] kthread+0x1d1/0x200 [ 2820.169895] ? set_kthread_struct+0x70/0x70 [ 2820.169899] ret_from_fork+0x1f/0x30 [ 2820.201447] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2820.217231] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2820.221297] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2820.225385] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2820.227748] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2820.243318] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2820.252873] RAX: 0000000000005608 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2820.258690] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2820.263875] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2820.265864] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2820.277061] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2820.282531] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2820.289347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2820.292335] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2820.296415] Call Trace: [ 2820.297625] [ 2820.298397] ? show_regs.cold.9+0x22/0x2f [ 2820.299628] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2820.302068] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2820.303756] ? watchdog+0x40/0x40 [ 2820.307128] ? hrtimer_interrupt+0x128/0x2c0 [ 2820.311702] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2820.316019] ? apic_timer_interrupt+0xf/0x20 [ 2820.317114] [ 2820.320136] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2820.325308] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2820.330522] do_raw_spin_lock+0x177/0x190 [ 2820.333543] _raw_spin_lock+0x21/0x30 [ 2820.336320] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2820.343363] ? do_raw_spin_unlock+0x75/0x190 [ 2820.346340] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2820.349145] ? do_wait_intr+0xf0/0xf0 [ 2820.350015] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2820.355639] mdc_reint+0x2d/0x110 [mdc] [ 2820.360047] mdc_unlink+0x658/0x870 [mdc] [ 2820.362515] lmv_unlink+0x34d/0xbd0 [lmv] [ 2820.365601] ll_unlink+0x24d/0x7f0 [lustre] [ 2820.368892] vfs_unlink+0x202/0x330 [ 2820.371135] do_unlinkat+0x2fa/0x460 [ 2820.373147] __x64_sys_unlink+0x37/0x50 [ 2820.374770] do_syscall_64+0xc1/0x440 [ 2820.375734] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2820.378379] RIP: 0033:0x7f83e5f5c10b [ 2820.380816] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2820.393479] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2820.396868] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2820.400108] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2820.404894] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2820.407346] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2820.410057] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2832.060340] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 2832.064783] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2832.103580] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2832.116349] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2832.125876] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2832.131969] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2832.146663] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2832.160989] RAX: 0000000000003e86 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2832.169449] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 2832.187783] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2832.194175] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 2832.205788] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 2832.214592] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 2832.217692] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2832.225610] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 2832.231766] Call Trace: [ 2832.232664] [ 2832.233451] ? show_regs.cold.9+0x22/0x2f [ 2832.239836] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2832.243127] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2832.247063] ? watchdog+0x40/0x40 [ 2832.250163] ? hrtimer_interrupt+0x128/0x2c0 [ 2832.252302] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2832.256378] ? apic_timer_interrupt+0xf/0x20 [ 2832.258702] [ 2832.262139] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2832.269409] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2832.277519] do_raw_spin_lock+0x177/0x190 [ 2832.284429] _raw_spin_lock+0x21/0x30 [ 2832.289099] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2832.297515] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2832.302712] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2832.307296] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2832.315108] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2832.320738] ? woken_wake_function+0x30/0x30 [ 2832.323768] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2832.331121] kthread+0x1d1/0x200 [ 2832.333096] ? set_kthread_struct+0x70/0x70 [ 2832.335311] ret_from_fork+0x1f/0x30 [ 2844.060201] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 2844.068170] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2844.107492] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2844.113009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2844.116954] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2844.118968] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2844.127874] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2844.135999] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2844.146533] RDX: 00000000000040c0 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2844.156285] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2844.164458] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2844.170826] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2844.174635] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2844.179047] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2844.182018] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2844.188949] Call Trace: [ 2844.190446] [ 2844.194979] ? show_regs.cold.9+0x22/0x2f [ 2844.203240] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2844.206751] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2844.211155] ? watchdog+0x40/0x40 [ 2844.213964] ? hrtimer_interrupt+0x128/0x2c0 [ 2844.216516] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2844.218940] ? apic_timer_interrupt+0xf/0x20 [ 2844.220738] [ 2844.221505] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2844.227656] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2844.233856] do_raw_spin_lock+0x177/0x190 [ 2844.236063] _raw_spin_lock+0x21/0x30 [ 2844.238238] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2844.240222] obd_mod_free+0x46/0x140 [ptlrpc] [ 2844.242037] mdc_commit_open+0x91/0xb0 [mdc] [ 2844.243475] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2844.245906] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2844.248932] after_reply+0xe2c/0x1780 [ptlrpc] [ 2844.251774] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2844.257539] ? __next_timer_interrupt+0x160/0x160 [ 2844.262759] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2844.266967] ? do_wait_intr+0xf0/0xf0 [ 2844.269135] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2844.271869] kthread+0x1d1/0x200 [ 2844.273076] ? set_kthread_struct+0x70/0x70 [ 2844.275394] ret_from_fork+0x1f/0x30 [ 2848.060517] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 2848.064809] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2848.070479] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) [ 2848.075862] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2848.075956] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2848.075964] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2848.075966] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2848.075980] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2848.075984] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2848.075988] RAX: 0000000000000c88 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2848.075991] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2848.075993] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2848.075995] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2848.075998] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2848.076001] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2848.076004] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2848.076006] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2848.076015] Call Trace: [ 2848.076021] [ 2848.076027] ? show_regs.cold.9+0x22/0x2f [ 2848.076037] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2848.076044] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2848.076049] ? watchdog+0x40/0x40 [ 2848.076054] ? hrtimer_interrupt+0x128/0x2c0 [ 2848.076058] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2848.076068] ? apic_timer_interrupt+0xf/0x20 [ 2848.076074] [ 2848.076077] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2848.076081] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2848.076086] do_raw_spin_lock+0x177/0x190 [ 2848.076090] _raw_spin_lock+0x21/0x30 [ 2848.076097] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2848.089121] fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2848.089174] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2848.089180] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2848.089186] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2848.173950] ? do_raw_spin_unlock+0x75/0x190 [ 2848.173974] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2848.174285] ? do_wait_intr+0xf0/0xf0 [ 2848.174303] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2848.174578] mdc_reint+0x2d/0x110 [mdc] [ 2848.174784] mdc_unlink+0x658/0x870 [mdc] [ 2848.174812] lmv_unlink+0x34d/0xbd0 [lmv] [ 2848.174849] ll_unlink+0x24d/0x7f0 [lustre] [ 2848.175038] vfs_unlink+0x202/0x330 [ 2848.175046] do_unlinkat+0x2fa/0x460 [ 2848.175051] __x64_sys_unlink+0x37/0x50 [ 2848.175054] do_syscall_64+0xc1/0x440 [ 2848.175061] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2848.175069] RIP: 0033:0x7f83e5f5c10b [ 2848.175076] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2848.175079] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2848.175085] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2848.175088] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2848.175090] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2848.175092] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2848.175094] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2848.569423] [ 2848.577244] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2848.580901] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2848.608433] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2848.618829] RAX: 0000000000001d03 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2848.624260] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2848.636834] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2848.649752] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2848.659720] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2848.668754] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2848.681907] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2848.684550] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2848.697627] Call Trace: [ 2848.699516] [ 2848.701498] ? show_regs.cold.9+0x22/0x2f [ 2848.708048] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2848.713431] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2848.717356] ? watchdog+0x40/0x40 [ 2848.725301] ? hrtimer_interrupt+0x128/0x2c0 [ 2848.728342] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2848.734809] ? apic_timer_interrupt+0xf/0x20 [ 2848.740678] [ 2848.741864] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2848.750133] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2848.756343] do_raw_spin_lock+0x177/0x190 [ 2848.761746] _raw_spin_lock+0x21/0x30 [ 2848.763700] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2848.767335] process_one_work+0x2c8/0x700 [ 2848.772357] worker_thread+0x296/0x6e0 [ 2848.775205] ? rescuer_thread+0x570/0x570 [ 2848.777147] kthread+0x1d1/0x200 [ 2848.778881] ? set_kthread_struct+0x70/0x70 [ 2848.780989] ret_from_fork+0x1f/0x30 [ 2849.769102] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 336s! [ 2849.786022] Showing busy workqueues and worker pools: [ 2849.801990] workqueue events: flags=0x0 [ 2849.808604] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2849.821441] pending: osc_grant_work_handler [osc], push_to_pool [ 2849.837954] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2849.846078] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 2849.866673] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2849.880011] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 2849.906278] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2849.918475] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 2849.927872] workqueue events_long: flags=0x0 [ 2849.936023] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2849.946884] pending: lu_objects_flush [obdclass] [ 2849.958245] workqueue events_unbound: flags=0x2 [ 2849.966249] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2849.972993] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2849.985285] workqueue events_power_efficient: flags=0x80 [ 2849.991058] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2850.002108] pending: neigh_periodic_work, do_cache_clean [ 2850.012354] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2850.019637] pending: neigh_periodic_work, check_lifetime [ 2850.027613] workqueue mm_percpu_wq: flags=0x8 [ 2850.034215] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2850.041461] pending: vmstat_update, lru_add_drain_per_cpu [ 2850.051209] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2850.067015] pending: vmstat_update [ 2850.081020] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2850.092111] pending: lru_add_drain_per_cpu BAR(48) [ 2850.100043] workqueue ipv6_addrconf: flags=0x40008 [ 2850.105020] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2850.125928] pending: addrconf_verify_work [ 2850.137185] workqueue knbd0-recv: flags=0x1a [ 2850.149697] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2850.157913] in-flight: 650:recv_work [ 2850.165283] workqueue ptlrpc_pinger: flags=0x2 [ 2850.166922] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2850.182868] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2850.186807] pool 8: cpus=0-3 flags=0x4 nice=0 hung=336s workers=4 idle: 36 466 29757 [ 2850.202769] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2840s workers=2 idle: 652 [ 2860.059036] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 2860.062161] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2860.108090] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2860.116540] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2860.119759] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2860.122380] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2860.134774] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2860.137474] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2860.142753] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 2860.146931] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2860.155344] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 2860.159062] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 2860.162537] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 2860.168929] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2860.176824] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 2860.183785] Call Trace: [ 2860.184952] [ 2860.186301] ? show_regs.cold.9+0x22/0x2f [ 2860.188285] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2860.191748] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2860.199591] ? watchdog+0x40/0x40 [ 2860.201641] ? hrtimer_interrupt+0x128/0x2c0 [ 2860.206341] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2860.209625] ? apic_timer_interrupt+0xf/0x20 [ 2860.214306] [ 2860.215051] ? native_safe_halt+0x2e/0x50 [ 2860.218665] kvm_wait+0xcd/0xe0 [ 2860.225666] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2860.232921] do_raw_spin_lock+0x177/0x190 [ 2860.238956] _raw_spin_lock+0x21/0x30 [ 2860.240154] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2860.247681] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2860.254315] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2860.260093] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2860.269977] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2860.277813] ? woken_wake_function+0x30/0x30 [ 2860.280929] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2860.288917] kthread+0x1d1/0x200 [ 2860.296779] ? set_kthread_struct+0x70/0x70 [ 2860.302069] ret_from_fork+0x1f/0x30 [ 2876.061436] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 2876.062381] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 2876.063280] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2876.063289] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2876.063359] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2876.063366] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2876.063369] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2876.065005] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2876.065011] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2876.065017] RAX: 0000000000007b86 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2876.065020] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2876.065022] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2876.065023] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2876.065025] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2876.065029] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2876.065031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2876.065033] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2876.065042] Call Trace: [ 2876.065048] [ 2876.065066] ? show_regs.cold.9+0x22/0x2f [ 2876.065078] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2876.065090] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2876.065098] ? watchdog+0x40/0x40 [ 2876.065101] ? hrtimer_interrupt+0x128/0x2c0 [ 2876.065105] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2876.065116] ? apic_timer_interrupt+0xf/0x20 [ 2876.065121] [ 2876.065125] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2876.065131] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2876.065135] do_raw_spin_lock+0x177/0x190 [ 2876.065140] _raw_spin_lock+0x21/0x30 [ 2876.065147] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2876.067928] ? do_raw_spin_unlock+0x75/0x190 [ 2876.067947] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2876.068289] ? do_wait_intr+0xf0/0xf0 [ 2876.068299] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2876.069022] mdc_reint+0x2d/0x110 [mdc] [ 2876.069076] mdc_unlink+0x658/0x870 [mdc] [ 2876.069105] lmv_unlink+0x34d/0xbd0 [lmv] [ 2876.069137] ll_unlink+0x24d/0x7f0 [lustre] [ 2876.069258] vfs_unlink+0x202/0x330 [ 2876.069267] do_unlinkat+0x2fa/0x460 [ 2876.069273] __x64_sys_unlink+0x37/0x50 [ 2876.069277] do_syscall_64+0xc1/0x440 [ 2876.069284] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2876.069294] RIP: 0033:0x7f83e5f5c10b [ 2876.069301] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2876.069305] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2876.069310] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2876.069312] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2876.069314] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2876.069316] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2876.069318] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2876.071070] Modules linked in: lustre(O) [ 2876.077246] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2876.077469] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2876.077475] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2876.077481] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2876.077858] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2876.077986] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2876.077991] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2876.077995] RAX: 0000000000004a09 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2876.077998] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2876.078000] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2876.078002] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2876.078005] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2876.078008] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2876.078011] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2876.078013] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2876.078021] Call Trace: [ 2876.078026] [ 2876.078031] ? show_regs.cold.9+0x22/0x2f [ 2876.078041] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2876.078047] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2876.078062] ? watchdog+0x40/0x40 [ 2876.078066] ? hrtimer_interrupt+0x128/0x2c0 [ 2876.078070] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2876.078082] ? apic_timer_interrupt+0xf/0x20 [ 2876.078087] [ 2876.078090] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2876.078095] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2876.078100] do_raw_spin_lock+0x177/0x190 [ 2876.078105] _raw_spin_lock+0x21/0x30 [ 2876.078110] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2876.078432] process_one_work+0x2c8/0x700 [ 2876.078439] worker_thread+0x296/0x6e0 [ 2876.078443] ? rescuer_thread+0x570/0x570 [ 2876.078446] kthread+0x1d1/0x200 [ 2876.078451] ? set_kthread_struct+0x70/0x70 [ 2876.078454] ret_from_fork+0x1f/0x30 [ 2876.494799] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2876.546652] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2876.560140] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2876.574645] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2876.578615] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2876.606330] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2876.614550] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2876.618215] RDX: 00000000000054a5 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2876.620880] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2876.627457] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2876.630337] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2876.632762] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2876.635470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2876.637096] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2876.641164] Call Trace: [ 2876.641932] [ 2876.642536] ? show_regs.cold.9+0x22/0x2f [ 2876.645967] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2876.649074] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2876.650740] ? watchdog+0x40/0x40 [ 2876.651796] ? hrtimer_interrupt+0x128/0x2c0 [ 2876.653571] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2876.655129] ? apic_timer_interrupt+0xf/0x20 [ 2876.657916] [ 2876.658805] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2876.660894] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2876.662867] do_raw_spin_lock+0x177/0x190 [ 2876.664450] _raw_spin_lock+0x21/0x30 [ 2876.665750] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2876.667918] obd_mod_free+0x46/0x140 [ptlrpc] [ 2876.669554] mdc_commit_open+0x91/0xb0 [mdc] [ 2876.670793] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2876.672787] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2876.674937] after_reply+0xe2c/0x1780 [ptlrpc] [ 2876.676703] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2876.679423] ? __next_timer_interrupt+0x160/0x160 [ 2876.680981] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2876.682950] ? do_wait_intr+0xf0/0xf0 [ 2876.684098] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2876.686550] kthread+0x1d1/0x200 [ 2876.687667] ? set_kthread_struct+0x70/0x70 [ 2876.689421] ret_from_fork+0x1f/0x30 [ 2880.487567] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 367s! [ 2880.506829] Showing busy workqueues and worker pools: [ 2880.511277] workqueue events: flags=0x0 [ 2880.520221] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2880.531073] pending: osc_grant_work_handler [osc], push_to_pool [ 2880.543768] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2880.558316] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 2880.570036] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2880.583794] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 2880.594541] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2880.607304] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 2880.614089] workqueue events_long: flags=0x0 [ 2880.619996] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2880.627953] pending: lu_objects_flush [obdclass] [ 2880.631873] workqueue events_unbound: flags=0x2 [ 2880.641093] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2880.650828] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2880.663367] workqueue events_power_efficient: flags=0x80 [ 2880.670693] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2880.683872] pending: neigh_periodic_work, do_cache_clean [ 2880.692726] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2880.704477] pending: neigh_periodic_work, check_lifetime [ 2880.710927] workqueue mm_percpu_wq: flags=0x8 [ 2880.712627] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2880.718862] pending: vmstat_update, lru_add_drain_per_cpu [ 2880.721380] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2880.723855] pending: vmstat_update [ 2880.725154] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2880.729232] pending: lru_add_drain_per_cpu BAR(48) [ 2880.735524] workqueue ipv6_addrconf: flags=0x40008 [ 2880.740736] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2880.743831] pending: addrconf_verify_work [ 2880.746027] workqueue knbd0-recv: flags=0x1a [ 2880.750940] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2880.755053] in-flight: 650:recv_work [ 2880.756480] workqueue ptlrpc_pinger: flags=0x2 [ 2880.757871] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2880.760807] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2880.763611] pool 8: cpus=0-3 flags=0x4 nice=0 hung=367s workers=4 idle: 36 466 29757 [ 2880.766550] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2870s workers=2 idle: 652 [ 2888.062270] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 2888.067702] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2888.136109] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2888.153512] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2888.165100] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2888.180674] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2888.203865] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2888.211766] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2888.219581] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 2888.222405] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2888.223916] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 2888.234577] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 2888.240194] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 2888.243692] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2888.251862] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 2888.259636] Call Trace: [ 2888.263435] [ 2888.264855] ? show_regs.cold.9+0x22/0x2f [ 2888.266585] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2888.270719] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2888.273236] ? watchdog+0x40/0x40 [ 2888.276544] ? hrtimer_interrupt+0x128/0x2c0 [ 2888.282670] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2888.290077] ? apic_timer_interrupt+0xf/0x20 [ 2888.297143] [ 2888.298930] ? native_safe_halt+0x2e/0x50 [ 2888.304632] kvm_wait+0xcd/0xe0 [ 2888.306246] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2888.315576] do_raw_spin_lock+0x177/0x190 [ 2888.327211] _raw_spin_lock+0x21/0x30 [ 2888.332668] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2888.335626] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2888.340048] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2888.348866] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2888.356645] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2888.363862] ? woken_wake_function+0x30/0x30 [ 2888.367735] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2888.375640] kthread+0x1d1/0x200 [ 2888.381522] ? set_kthread_struct+0x70/0x70 [ 2888.387747] ret_from_fork+0x1f/0x30 [ 2904.060823] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 2904.062030] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2904.067164] Modules linked in: [ 2904.070091] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2904.070192] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2904.070199] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2904.070202] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2904.070219] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2904.076175] lustre(O) [ 2904.100231] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2904.100317] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2904.100320] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2904.100322] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2904.100324] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2904.100326] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2904.100329] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2904.100332] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2904.100334] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2904.100341] Call Trace: [ 2904.100348] [ 2904.100353] ? show_regs.cold.9+0x22/0x2f [ 2904.100366] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2904.100373] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2904.100377] ? watchdog+0x40/0x40 [ 2904.100381] ? hrtimer_interrupt+0x128/0x2c0 [ 2904.100385] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2904.100392] ? apic_timer_interrupt+0xf/0x20 [ 2904.100397] [ 2904.100400] ? native_safe_halt+0x2e/0x50 [ 2904.100405] kvm_wait+0xcd/0xe0 [ 2904.100411] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2904.100418] do_raw_spin_lock+0x177/0x190 [ 2904.100422] _raw_spin_lock+0x21/0x30 [ 2904.100429] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2904.100790] ? do_raw_spin_unlock+0x75/0x190 [ 2904.100796] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2904.101049] ? do_wait_intr+0xf0/0xf0 [ 2904.101056] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2904.101253] mdc_reint+0x2d/0x110 [mdc] [ 2904.101289] mdc_unlink+0x658/0x870 [mdc] [ 2904.101318] lmv_unlink+0x34d/0xbd0 [lmv] [ 2904.101348] ll_unlink+0x24d/0x7f0 [lustre] [ 2904.101458] vfs_unlink+0x202/0x330 [ 2904.101468] do_unlinkat+0x2fa/0x460 [ 2904.101473] __x64_sys_unlink+0x37/0x50 [ 2904.101476] do_syscall_64+0xc1/0x440 [ 2904.101483] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2904.101489] RIP: 0033:0x7f83e5f5c10b [ 2904.101496] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2904.101500] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2904.101504] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2904.101506] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2904.101508] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2904.101510] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2904.101513] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2904.252488] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2904.277088] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2904.281539] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2904.285203] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2904.287483] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2904.289160] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2904.295587] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2904.298815] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2904.302005] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2904.304950] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2904.308572] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2904.320812] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2904.328512] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2904.333907] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2904.342812] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2904.349128] Call Trace: [ 2904.352289] [ 2904.354451] ? show_regs.cold.9+0x22/0x2f [ 2904.358197] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2904.364914] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2904.368571] ? watchdog+0x40/0x40 [ 2904.369826] ? hrtimer_interrupt+0x128/0x2c0 [ 2904.371997] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2904.373702] ? apic_timer_interrupt+0xf/0x20 [ 2904.376358] [ 2904.377298] ? native_safe_halt+0x2e/0x50 [ 2904.380161] kvm_wait+0xcd/0xe0 [ 2904.385386] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2904.393901] do_raw_spin_lock+0x177/0x190 [ 2904.406262] _raw_spin_lock+0x21/0x30 [ 2904.409659] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2904.411590] process_one_work+0x2c8/0x700 [ 2904.418905] worker_thread+0x296/0x6e0 [ 2904.423959] ? rescuer_thread+0x570/0x570 [ 2904.428654] kthread+0x1d1/0x200 [ 2904.430660] ? set_kthread_struct+0x70/0x70 [ 2904.433904] ret_from_fork+0x1f/0x30 [ 2908.056014] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 2908.059907] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2908.079493] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2908.084511] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2908.090389] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2908.093826] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2908.103311] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2908.107019] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2908.110862] RDX: 00000000000055c7 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2908.114528] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2908.118198] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2908.121491] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2908.125507] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2908.129354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2908.131820] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2908.135381] Call Trace: [ 2908.136483] [ 2908.137363] ? show_regs.cold.9+0x22/0x2f [ 2908.139832] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2908.141921] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2908.143963] ? watchdog+0x40/0x40 [ 2908.145349] ? hrtimer_interrupt+0x128/0x2c0 [ 2908.147407] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2908.149937] ? apic_timer_interrupt+0xf/0x20 [ 2908.152249] [ 2908.153163] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2908.155493] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2908.158238] do_raw_spin_lock+0x177/0x190 [ 2908.159883] _raw_spin_lock+0x21/0x30 [ 2908.162492] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2908.165469] obd_mod_free+0x46/0x140 [ptlrpc] [ 2908.168559] mdc_commit_open+0x91/0xb0 [mdc] [ 2908.172250] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2908.177100] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2908.181294] after_reply+0xe2c/0x1780 [ptlrpc] [ 2908.184191] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2908.186861] ? __next_timer_interrupt+0x160/0x160 [ 2908.188621] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2908.190650] ? do_wait_intr+0xf0/0xf0 [ 2908.191710] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2908.194039] kthread+0x1d1/0x200 [ 2908.194978] ? set_kthread_struct+0x70/0x70 [ 2908.196131] ret_from_fork+0x1f/0x30 [ 2911.200438] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 398s! [ 2911.203280] Showing busy workqueues and worker pools: [ 2911.204248] workqueue events: flags=0x0 [ 2911.205308] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2911.206755] pending: osc_grant_work_handler [osc], push_to_pool [ 2911.208330] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2911.209776] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 2911.211421] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2911.213040] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 2911.215011] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2911.216582] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 2911.218708] workqueue events_long: flags=0x0 [ 2911.219693] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2911.221256] pending: lu_objects_flush [obdclass] [ 2911.222517] workqueue events_unbound: flags=0x2 [ 2911.223596] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2911.225191] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2911.226601] workqueue events_power_efficient: flags=0x80 [ 2911.227863] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2911.229512] pending: neigh_periodic_work, do_cache_clean [ 2911.230761] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2911.232384] pending: neigh_periodic_work, check_lifetime [ 2911.233594] workqueue mm_percpu_wq: flags=0x8 [ 2911.234579] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2911.236174] pending: vmstat_update, lru_add_drain_per_cpu [ 2911.237459] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2911.239013] pending: vmstat_update [ 2911.240137] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2911.242370] pending: lru_add_drain_per_cpu BAR(48) [ 2911.244382] workqueue ipv6_addrconf: flags=0x40008 [ 2911.245982] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2911.248817] pending: addrconf_verify_work [ 2911.250670] workqueue knbd0-recv: flags=0x1a [ 2911.252354] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2911.254733] in-flight: 650:recv_work [ 2911.255819] workqueue ptlrpc_pinger: flags=0x2 [ 2911.257098] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2911.258679] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2911.260482] pool 8: cpus=0-3 flags=0x4 nice=0 hung=398s workers=4 idle: 36 466 29757 [ 2911.263727] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2901s workers=2 idle: 652 [ 2916.059528] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 2916.064423] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2916.092764] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2916.098358] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2916.103001] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2916.106196] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2916.116295] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2916.119840] RAX: 0000000000000711 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2916.123298] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 2916.126544] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2916.131659] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 2916.135838] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 2916.138435] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 2916.142206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2916.144779] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 2916.148311] Call Trace: [ 2916.149650] [ 2916.150555] ? show_regs.cold.9+0x22/0x2f [ 2916.152336] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2916.154746] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2916.157083] ? watchdog+0x40/0x40 [ 2916.158492] ? hrtimer_interrupt+0x128/0x2c0 [ 2916.161608] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2916.166832] ? apic_timer_interrupt+0xf/0x20 [ 2916.169345] [ 2916.170514] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2916.173282] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2916.175656] do_raw_spin_lock+0x177/0x190 [ 2916.177821] _raw_spin_lock+0x21/0x30 [ 2916.179499] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2916.182573] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2916.185535] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2916.188528] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2916.191459] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2916.194197] ? woken_wake_function+0x30/0x30 [ 2916.196287] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2916.198687] kthread+0x1d1/0x200 [ 2916.200217] ? set_kthread_struct+0x70/0x70 [ 2916.202036] ret_from_fork+0x1f/0x30 [ 2932.061552] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 2932.062018] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2932.066220] Modules linked in: lustre(O) [ 2932.068842] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2932.068920] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2932.068927] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2932.068931] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2932.068943] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2932.068948] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2932.068952] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2932.068954] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2932.068957] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2932.068959] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2932.068961] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2932.068965] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2932.068967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2932.068970] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2932.068977] Call Trace: [ 2932.068982] [ 2932.068987] ? show_regs.cold.9+0x22/0x2f [ 2932.068995] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2932.069001] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2932.069006] ? watchdog+0x40/0x40 [ 2932.069010] ? hrtimer_interrupt+0x128/0x2c0 [ 2932.069014] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2932.069020] ? apic_timer_interrupt+0xf/0x20 [ 2932.069024] [ 2932.069027] ? native_safe_halt+0x2e/0x50 [ 2932.069031] kvm_wait+0xcd/0xe0 [ 2932.069037] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2932.069044] do_raw_spin_lock+0x177/0x190 [ 2932.069048] _raw_spin_lock+0x21/0x30 [ 2932.069052] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2932.069332] ? do_raw_spin_unlock+0x75/0x190 [ 2932.069338] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2932.069759] ? do_wait_intr+0xf0/0xf0 [ 2932.069767] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2932.069970] mdc_reint+0x2d/0x110 [mdc] [ 2932.070008] mdc_unlink+0x658/0x870 [mdc] [ 2932.070042] lmv_unlink+0x34d/0xbd0 [lmv] [ 2932.070073] ll_unlink+0x24d/0x7f0 [lustre] [ 2932.070198] vfs_unlink+0x202/0x330 [ 2932.070206] do_unlinkat+0x2fa/0x460 [ 2932.070210] __x64_sys_unlink+0x37/0x50 [ 2932.070213] do_syscall_64+0xc1/0x440 [ 2932.070219] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2932.070226] RIP: 0033:0x7f83e5f5c10b [ 2932.070232] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2932.070236] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2932.070241] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2932.070243] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2932.070246] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2932.070248] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2932.070250] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2932.460924] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2932.487708] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2932.495621] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2932.504859] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2932.509587] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2932.512384] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2932.523960] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2932.528299] RAX: 000000000000597c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2932.533107] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2932.540968] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2932.547317] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2932.550692] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2932.554599] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2932.562191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2932.566105] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2932.570528] Call Trace: [ 2932.571676] [ 2932.573190] ? show_regs.cold.9+0x22/0x2f [ 2932.584891] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2932.593323] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2932.595495] ? watchdog+0x40/0x40 [ 2932.599456] ? hrtimer_interrupt+0x128/0x2c0 [ 2932.601597] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2932.605035] ? apic_timer_interrupt+0xf/0x20 [ 2932.610398] [ 2932.613458] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2932.616635] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2932.622816] do_raw_spin_lock+0x177/0x190 [ 2932.624357] _raw_spin_lock+0x21/0x30 [ 2932.629067] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2932.636986] process_one_work+0x2c8/0x700 [ 2932.641599] worker_thread+0x296/0x6e0 [ 2932.643660] ? rescuer_thread+0x570/0x570 [ 2932.649094] kthread+0x1d1/0x200 [ 2932.651382] ? set_kthread_struct+0x70/0x70 [ 2932.656412] ret_from_fork+0x1f/0x30 [ 2936.057242] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 2936.068480] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2936.112309] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2936.117310] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2936.123321] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2936.127327] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2936.137945] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2936.142013] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2936.146065] RDX: 00000000000038b9 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2936.149652] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2936.153314] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2936.156967] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2936.160466] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2936.164483] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2936.167492] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2936.171051] Call Trace: [ 2936.172346] [ 2936.173414] ? show_regs.cold.9+0x22/0x2f [ 2936.175400] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2936.178516] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2936.180927] ? watchdog+0x40/0x40 [ 2936.182459] ? hrtimer_interrupt+0x128/0x2c0 [ 2936.184562] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2936.187100] ? apic_timer_interrupt+0xf/0x20 [ 2936.189596] [ 2936.190845] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2936.193556] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2936.196249] do_raw_spin_lock+0x177/0x190 [ 2936.198239] _raw_spin_lock+0x21/0x30 [ 2936.200001] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2936.202005] obd_mod_free+0x46/0x140 [ptlrpc] [ 2936.203960] mdc_commit_open+0x91/0xb0 [mdc] [ 2936.205428] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2936.208192] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2936.211192] after_reply+0xe2c/0x1780 [ptlrpc] [ 2936.213518] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2936.216436] ? __next_timer_interrupt+0x160/0x160 [ 2936.218889] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2936.221208] ? do_wait_intr+0xf0/0xf0 [ 2936.222991] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2936.225013] kthread+0x1d1/0x200 [ 2936.226085] ? set_kthread_struct+0x70/0x70 [ 2936.228197] ret_from_fork+0x1f/0x30 [ 2940.976159] rcu: INFO: rcu_sched self-detected stall on CPU [ 2940.980541] rcu: 0-....: (624158 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 2940.990931] (t=960031 jiffies g=262441 q=57224) [ 2940.997545] rcu: rcu_sched kthread starved for 429225 jiffies! g262441 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 2941.007148] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2941.017557] rcu: RCU grace-period kthread stack dump: [ 2941.024254] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2941.035404] Call Trace: [ 2941.039516] __schedule+0x351/0xcb0 [ 2941.041509] schedule+0xc0/0x180 [ 2941.044822] schedule_timeout+0xb4/0x190 [ 2941.047568] ? do_raw_spin_unlock+0x75/0x190 [ 2941.050370] ? __next_timer_interrupt+0x160/0x160 [ 2941.053483] ? prepare_to_swait_event+0xf3/0x140 [ 2941.056485] rcu_gp_kthread+0x8dd/0x15b0 [ 2941.057646] ? force_qs_rnp+0x2c0/0x2c0 [ 2941.063957] kthread+0x1d1/0x200 [ 2941.065559] ? set_kthread_struct+0x70/0x70 [ 2941.069809] ret_from_fork+0x1f/0x30 [ 2941.071803] rcu: Stack dump where RCU GP kthread last ran: [ 2941.078299] Sending NMI from CPU 0 to CPUs 2: [ 2941.080660] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [ 2941.084108] NMI backtrace for cpu 0 [ 2941.086162] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2941.096913] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2941.107960] Call Trace: [ 2941.109118] [ 2941.110918] ? dump_stack+0xbb/0x10e [ 2941.119602] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2941.127287] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2941.129474] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2941.135811] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2941.142988] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2941.149128] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2941.151496] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2941.154485] ? update_process_times+0x6d/0xc0 [ 2941.159241] ? tick_sched_handle.isra.18+0x27/0x90 [ 2941.162831] ? tick_sched_timer+0x53/0xb0 [ 2941.168659] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2941.171864] ? tick_sched_handle.isra.18+0x90/0x90 [ 2941.174094] ? hrtimer_interrupt+0x128/0x2c0 [ 2941.176261] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2941.178405] ? apic_timer_interrupt+0xf/0x20 [ 2941.180475] [ 2941.181339] ? native_safe_halt+0x2e/0x50 [ 2941.187858] ? kvm_wait+0xcd/0xe0 [ 2941.189462] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2941.192760] ? do_raw_spin_lock+0x177/0x190 [ 2941.197071] ? _raw_spin_lock+0x21/0x30 [ 2941.199137] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2941.202252] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2941.206288] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2941.209073] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2941.212318] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2941.214352] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2941.219383] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2941.222377] ? __next_timer_interrupt+0x160/0x160 [ 2941.224775] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2941.229145] ? do_wait_intr+0xf0/0xf0 [ 2941.231188] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2941.234058] ? kthread+0x1d1/0x200 [ 2941.235699] ? set_kthread_struct+0x70/0x70 [ 2941.237575] ? ret_from_fork+0x1f/0x30 [ 2941.410030] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 428s! [ 2941.416338] Showing busy workqueues and worker pools: [ 2941.419518] workqueue events: flags=0x0 [ 2941.421128] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2941.424238] pending: osc_grant_work_handler [osc], push_to_pool [ 2941.427478] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2941.431123] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 2941.434132] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2941.437770] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 2941.442282] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2941.445762] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 2941.449689] workqueue events_long: flags=0x0 [ 2941.452121] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2941.455119] pending: lu_objects_flush [obdclass] [ 2941.457806] workqueue events_unbound: flags=0x2 [ 2941.460558] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2941.464134] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2941.466869] workqueue events_power_efficient: flags=0x80 [ 2941.469349] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2941.474117] pending: neigh_periodic_work, do_cache_clean [ 2941.476906] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2941.480206] pending: neigh_periodic_work, check_lifetime [ 2941.483306] workqueue mm_percpu_wq: flags=0x8 [ 2941.485247] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2941.487727] pending: vmstat_update, lru_add_drain_per_cpu [ 2941.490981] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2941.494265] pending: vmstat_update [ 2941.495946] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2941.498451] pending: lru_add_drain_per_cpu BAR(48) [ 2941.500985] workqueue ipv6_addrconf: flags=0x40008 [ 2941.503891] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2941.507563] pending: addrconf_verify_work [ 2941.510697] workqueue knbd0-recv: flags=0x1a [ 2941.513266] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2941.516621] in-flight: 650:recv_work [ 2941.518510] workqueue ptlrpc_pinger: flags=0x2 [ 2941.521264] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2941.524323] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2941.527072] pool 8: cpus=0-3 flags=0x4 nice=0 hung=428s workers=4 idle: 36 466 29757 [ 2941.531092] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2931s workers=2 idle: 652 [ 2944.067030] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 2944.079874] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2944.153713] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2944.171201] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2944.198744] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2944.213857] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2944.257195] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2944.259935] RAX: 0000000000002909 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2944.273049] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 2944.283502] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2944.291059] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 2944.302847] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 2944.310308] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 2944.315093] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2944.319351] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 2944.326835] Call Trace: [ 2944.330101] [ 2944.332357] ? show_regs.cold.9+0x22/0x2f [ 2944.339538] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2944.341384] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2944.348033] ? watchdog+0x40/0x40 [ 2944.354762] ? hrtimer_interrupt+0x128/0x2c0 [ 2944.358296] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2944.363477] ? apic_timer_interrupt+0xf/0x20 [ 2944.367787] [ 2944.369033] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2944.374042] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2944.381046] do_raw_spin_lock+0x177/0x190 [ 2944.384096] _raw_spin_lock+0x21/0x30 [ 2944.387477] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2944.390608] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2944.398916] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2944.406379] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2944.413498] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2944.416772] ? woken_wake_function+0x30/0x30 [ 2944.418321] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2944.425676] kthread+0x1d1/0x200 [ 2944.429659] ? set_kthread_struct+0x70/0x70 [ 2944.433946] ret_from_fork+0x1f/0x30 [ 2960.062079] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2960.067348] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2960.090185] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2960.097105] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2960.101567] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2960.104951] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2960.114486] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2960.118338] RAX: 0000000000003067 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2960.121037] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2960.124414] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2960.127655] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2960.131458] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2960.135096] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2960.138377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2960.142042] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2960.145143] Call Trace: [ 2960.146369] [ 2960.147372] ? show_regs.cold.9+0x22/0x2f [ 2960.149784] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2960.152334] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2960.154599] ? watchdog+0x40/0x40 [ 2960.156176] ? hrtimer_interrupt+0x128/0x2c0 [ 2960.158517] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2960.160736] ? apic_timer_interrupt+0xf/0x20 [ 2960.163187] [ 2960.164310] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2960.167340] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2960.170905] do_raw_spin_lock+0x177/0x190 [ 2960.172882] _raw_spin_lock+0x21/0x30 [ 2960.175529] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2960.179516] ? do_raw_spin_unlock+0x75/0x190 [ 2960.182123] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2960.185139] ? do_wait_intr+0xf0/0xf0 [ 2960.187701] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2960.190795] mdc_reint+0x2d/0x110 [mdc] [ 2960.193262] mdc_unlink+0x658/0x870 [mdc] [ 2960.195656] lmv_unlink+0x34d/0xbd0 [lmv] [ 2960.197856] ll_unlink+0x24d/0x7f0 [lustre] [ 2960.201036] vfs_unlink+0x202/0x330 [ 2960.202858] do_unlinkat+0x2fa/0x460 [ 2960.206034] __x64_sys_unlink+0x37/0x50 [ 2960.208516] do_syscall_64+0xc1/0x440 [ 2960.210372] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2960.213409] RIP: 0033:0x7f83e5f5c10b [ 2960.215395] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2960.226777] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2960.231279] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2960.235749] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2960.239831] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2960.244218] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2960.248357] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2968.058264] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 2968.060094] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 2968.061087] Modules linked in: lustre(O) osp(O) ofd(O) [ 2968.064254] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2968.064333] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2968.064340] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2968.064347] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2968.067977] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2968.068046] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2968.068054] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2968.068057] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2968.068070] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2968.068075] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2968.068079] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2968.068081] RDX: 000000000000436c RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2968.068083] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2968.068086] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2968.068088] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2968.068091] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2968.068093] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2968.068095] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2968.068105] Call Trace: [ 2968.068110] [ 2968.068116] ? show_regs.cold.9+0x22/0x2f [ 2968.068126] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2968.068133] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2968.068137] ? watchdog+0x40/0x40 [ 2968.068141] ? hrtimer_interrupt+0x128/0x2c0 [ 2968.068145] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2968.068171] ? apic_timer_interrupt+0xf/0x20 [ 2968.068176] [ 2968.068178] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2968.068183] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2968.068188] do_raw_spin_lock+0x177/0x190 [ 2968.068193] _raw_spin_lock+0x21/0x30 [ 2968.068199] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2968.068476] obd_mod_free+0x46/0x140 [ptlrpc] [ 2968.068675] mdc_commit_open+0x91/0xb0 [mdc] [ 2968.068714] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2968.068913] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2968.069111] after_reply+0xe2c/0x1780 [ptlrpc] [ 2968.070901] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2968.071103] ? __next_timer_interrupt+0x160/0x160 [ 2968.071114] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2968.071483] ? do_wait_intr+0xf0/0xf0 [ 2968.071493] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2968.071696] kthread+0x1d1/0x200 [ 2968.071702] ? set_kthread_struct+0x70/0x70 [ 2968.071705] ret_from_fork+0x1f/0x30 [ 2968.365957] [ 2968.373034] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2968.378686] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2968.401807] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2968.411761] RAX: 0000000000007582 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2968.415664] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2968.418898] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2968.422914] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2968.428588] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2968.432669] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2968.436629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2968.438371] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2968.443265] Call Trace: [ 2968.445700] [ 2968.451384] ? show_regs.cold.9+0x22/0x2f [ 2968.453135] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2968.455877] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2968.457699] ? watchdog+0x40/0x40 [ 2968.460913] ? hrtimer_interrupt+0x128/0x2c0 [ 2968.462711] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2968.466219] ? apic_timer_interrupt+0xf/0x20 [ 2968.468003] [ 2968.468916] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2968.471477] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2968.473957] do_raw_spin_lock+0x177/0x190 [ 2968.475796] _raw_spin_lock+0x21/0x30 [ 2968.477648] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2968.479814] process_one_work+0x2c8/0x700 [ 2968.481783] worker_thread+0x296/0x6e0 [ 2968.483255] ? rescuer_thread+0x570/0x570 [ 2968.484769] kthread+0x1d1/0x200 [ 2968.485758] ? set_kthread_struct+0x70/0x70 [ 2968.487083] ret_from_fork+0x1f/0x30 [ 2971.617205] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 458s! [ 2971.619959] Showing busy workqueues and worker pools: [ 2971.621777] workqueue events: flags=0x0 [ 2971.623107] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2971.625405] pending: osc_grant_work_handler [osc], push_to_pool [ 2971.628067] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2971.631222] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 2971.635450] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2971.640693] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 2971.643604] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2971.646949] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 2971.651699] workqueue events_long: flags=0x0 [ 2971.653943] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2971.656935] pending: lu_objects_flush [obdclass] [ 2971.660718] workqueue events_unbound: flags=0x2 [ 2971.663031] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2971.669633] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2971.675234] workqueue events_power_efficient: flags=0x80 [ 2971.678032] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2971.681631] pending: neigh_periodic_work, do_cache_clean [ 2971.684682] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2971.695558] pending: neigh_periodic_work, check_lifetime [ 2971.701786] workqueue mm_percpu_wq: flags=0x8 [ 2971.704217] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2971.710557] pending: vmstat_update, lru_add_drain_per_cpu [ 2971.717648] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2971.724217] pending: vmstat_update [ 2971.732034] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 2971.740298] pending: lru_add_drain_per_cpu BAR(48) [ 2971.743188] workqueue ipv6_addrconf: flags=0x40008 [ 2971.749565] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2971.752076] pending: addrconf_verify_work [ 2971.762889] workqueue knbd0-recv: flags=0x1a [ 2971.774385] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2971.784546] in-flight: 650:recv_work [ 2971.789539] workqueue ptlrpc_pinger: flags=0x2 [ 2971.799813] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2971.807935] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 2971.811940] pool 8: cpus=0-3 flags=0x4 nice=0 hung=458s workers=4 idle: 36 466 29757 [ 2971.820438] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2961s workers=2 idle: 652 [ 2972.059015] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 2972.062579] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2972.095489] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2972.107552] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2972.114069] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2972.122179] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2972.130705] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2972.137720] RAX: 00000000000014d0 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2972.144024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 2972.153616] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2972.159944] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 2972.164504] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 2972.168219] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 2972.173908] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2972.178676] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 2972.183740] Call Trace: [ 2972.185186] [ 2972.185987] ? show_regs.cold.9+0x22/0x2f [ 2972.187804] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2972.190554] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2972.192823] ? watchdog+0x40/0x40 [ 2972.194711] ? hrtimer_interrupt+0x128/0x2c0 [ 2972.198218] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2972.201416] ? apic_timer_interrupt+0xf/0x20 [ 2972.206038] [ 2972.206897] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2972.210655] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2972.215937] do_raw_spin_lock+0x177/0x190 [ 2972.221401] _raw_spin_lock+0x21/0x30 [ 2972.224718] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2972.231420] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2972.233875] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2972.236544] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2972.238845] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2972.241426] ? woken_wake_function+0x30/0x30 [ 2972.243106] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2972.245652] kthread+0x1d1/0x200 [ 2972.246863] ? set_kthread_struct+0x70/0x70 [ 2972.248420] ret_from_fork+0x1f/0x30 [ 2988.062056] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 2988.066181] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2988.103312] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2988.111393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2988.117373] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2988.120762] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2988.132169] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2988.136871] RAX: 0000000000003c10 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2988.140453] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 2988.143834] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2988.147699] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 2988.151355] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 2988.154864] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 2988.159197] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2988.161465] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 2988.164901] Call Trace: [ 2988.165812] [ 2988.166780] ? show_regs.cold.9+0x22/0x2f [ 2988.168508] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2988.170523] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2988.173196] ? watchdog+0x40/0x40 [ 2988.174674] ? hrtimer_interrupt+0x128/0x2c0 [ 2988.176697] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2988.179437] ? apic_timer_interrupt+0xf/0x20 [ 2988.181556] [ 2988.182525] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2988.194838] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2988.198826] do_raw_spin_lock+0x177/0x190 [ 2988.209108] _raw_spin_lock+0x21/0x30 [ 2988.215596] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2988.219398] ? do_raw_spin_unlock+0x75/0x190 [ 2988.224544] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2988.236623] ? do_wait_intr+0xf0/0xf0 [ 2988.239462] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2988.245111] mdc_reint+0x2d/0x110 [mdc] [ 2988.251061] mdc_unlink+0x658/0x870 [mdc] [ 2988.253672] lmv_unlink+0x34d/0xbd0 [lmv] [ 2988.259353] ll_unlink+0x24d/0x7f0 [lustre] [ 2988.262261] vfs_unlink+0x202/0x330 [ 2988.264186] do_unlinkat+0x2fa/0x460 [ 2988.265824] __x64_sys_unlink+0x37/0x50 [ 2988.269773] do_syscall_64+0xc1/0x440 [ 2988.272639] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2988.275350] RIP: 0033:0x7f83e5f5c10b [ 2988.277169] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 2988.301686] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2988.308022] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 2988.316681] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 2988.325479] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 2988.333908] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 2988.342842] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2996.056101] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 2996.060024] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 2996.060678] Modules linked in: lustre(O) osp(O) ofd(O) [ 2996.063553] Modules linked in: [ 2996.063555] lod(O) mdt(O) mdd(O) [ 2996.067171] lustre(O) [ 2996.068467] mgs(O) lquota(O) [ 2996.069913] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2996.069982] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2996.069989] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2996.069995] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2996.070355] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2996.070365] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2996.070369] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2996.070373] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 2996.070375] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 2996.070378] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 2996.070380] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 2996.070382] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 2996.070385] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 2996.070388] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2996.070391] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 2996.070398] Call Trace: [ 2996.070404] [ 2996.070409] ? show_regs.cold.9+0x22/0x2f [ 2996.070418] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2996.070424] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2996.070428] ? watchdog+0x40/0x40 [ 2996.070433] ? hrtimer_interrupt+0x128/0x2c0 [ 2996.070437] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2996.070444] ? apic_timer_interrupt+0xf/0x20 [ 2996.070448] [ 2996.070452] ? native_safe_halt+0x2e/0x50 [ 2996.070456] kvm_wait+0xcd/0xe0 [ 2996.070463] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2996.070471] do_raw_spin_lock+0x177/0x190 [ 2996.070475] _raw_spin_lock+0x21/0x30 [ 2996.070480] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2996.070797] process_one_work+0x2c8/0x700 [ 2996.070805] worker_thread+0x296/0x6e0 [ 2996.070810] ? rescuer_thread+0x570/0x570 [ 2996.070814] kthread+0x1d1/0x200 [ 2996.070819] ? set_kthread_struct+0x70/0x70 [ 2996.070822] ret_from_fork+0x1f/0x30 [ 2996.249256] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2996.281016] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2996.291852] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2996.302470] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2996.303926] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2996.309878] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2996.312306] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 2996.314803] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 2996.317511] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 2996.320155] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 2996.323551] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2996.327431] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 2996.330441] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2996.333212] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 2996.336870] Call Trace: [ 2996.339330] [ 2996.340382] ? show_regs.cold.9+0x22/0x2f [ 2996.342520] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2996.344968] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2996.347600] ? watchdog+0x40/0x40 [ 2996.349249] ? hrtimer_interrupt+0x128/0x2c0 [ 2996.351949] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2996.355076] ? apic_timer_interrupt+0xf/0x20 [ 2996.357548] [ 2996.358770] ? native_safe_halt+0x2e/0x50 [ 2996.360679] kvm_wait+0xcd/0xe0 [ 2996.362268] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2996.365500] do_raw_spin_lock+0x177/0x190 [ 2996.367596] _raw_spin_lock+0x21/0x30 [ 2996.369370] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2996.372354] obd_mod_free+0x46/0x140 [ptlrpc] [ 2996.375066] mdc_commit_open+0x91/0xb0 [mdc] [ 2996.377462] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2996.379824] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2996.383384] after_reply+0xe2c/0x1780 [ptlrpc] [ 2996.385860] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2996.389186] ? __next_timer_interrupt+0x160/0x160 [ 2996.391484] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2996.394682] ? do_wait_intr+0xf0/0xf0 [ 2996.398061] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2996.401724] kthread+0x1d1/0x200 [ 2996.404485] ? set_kthread_struct+0x70/0x70 [ 2996.406640] ret_from_fork+0x1f/0x30 [ 3000.059026] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 3000.065612] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3000.102065] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3000.109206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3000.115450] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3000.118364] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3000.129358] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3000.134879] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3000.139117] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3000.143683] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3000.148863] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3000.154227] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3000.158126] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3000.162982] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3000.166408] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3000.169989] Call Trace: [ 3000.171366] [ 3000.172375] ? show_regs.cold.9+0x22/0x2f [ 3000.174575] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3000.177571] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3000.180324] ? watchdog+0x40/0x40 [ 3000.182425] ? hrtimer_interrupt+0x128/0x2c0 [ 3000.185060] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3000.188398] ? apic_timer_interrupt+0xf/0x20 [ 3000.191128] [ 3000.192308] ? native_safe_halt+0x2e/0x50 [ 3000.194191] kvm_wait+0xcd/0xe0 [ 3000.195999] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3000.200113] do_raw_spin_lock+0x177/0x190 [ 3000.203630] _raw_spin_lock+0x21/0x30 [ 3000.205745] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3000.210230] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3000.214204] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3000.217541] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3000.220906] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3000.223690] ? woken_wake_function+0x30/0x30 [ 3000.226965] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3000.231082] kthread+0x1d1/0x200 [ 3000.233446] ? set_kthread_struct+0x70/0x70 [ 3000.236230] ret_from_fork+0x1f/0x30 [ 3002.339149] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 489s! [ 3002.349331] Showing busy workqueues and worker pools: [ 3002.357323] workqueue events: flags=0x0 [ 3002.362727] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3002.372745] pending: osc_grant_work_handler [osc], push_to_pool [ 3002.386220] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3002.395983] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3002.403845] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3002.411720] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3002.424734] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3002.430976] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3002.443714] workqueue events_long: flags=0x0 [ 3002.450160] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3002.457537] pending: lu_objects_flush [obdclass] [ 3002.460970] workqueue events_unbound: flags=0x2 [ 3002.467574] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3002.475517] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3002.484745] workqueue events_power_efficient: flags=0x80 [ 3002.492372] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3002.495237] pending: neigh_periodic_work, do_cache_clean [ 3002.496961] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3002.500754] pending: neigh_periodic_work, check_lifetime [ 3002.506677] workqueue mm_percpu_wq: flags=0x8 [ 3002.511221] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3002.518816] pending: vmstat_update, lru_add_drain_per_cpu [ 3002.524156] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3002.531950] pending: vmstat_update [ 3002.535179] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3002.547554] pending: lru_add_drain_per_cpu BAR(48) [ 3002.551212] workqueue ipv6_addrconf: flags=0x40008 [ 3002.556567] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3002.563974] pending: addrconf_verify_work [ 3002.567715] workqueue knbd0-recv: flags=0x1a [ 3002.569660] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3002.572979] in-flight: 650:recv_work [ 3002.575215] workqueue ptlrpc_pinger: flags=0x2 [ 3002.577396] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3002.581256] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3002.590260] pool 8: cpus=0-3 flags=0x4 nice=0 hung=489s workers=4 idle: 36 466 29757 [ 3002.594261] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2992s workers=2 idle: 652 [ 3016.065369] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 3016.070352] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3016.095970] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3016.102081] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3016.107279] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3016.111293] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3016.122752] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3016.128199] RAX: 0000000000005643 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3016.132236] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3016.136351] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3016.140687] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3016.145032] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3016.152218] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3016.159056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3016.164039] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3016.172689] Call Trace: [ 3016.181207] [ 3016.184741] ? show_regs.cold.9+0x22/0x2f [ 3016.191956] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3016.195287] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3016.206270] ? watchdog+0x40/0x40 [ 3016.207413] ? hrtimer_interrupt+0x128/0x2c0 [ 3016.210735] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3016.212603] ? apic_timer_interrupt+0xf/0x20 [ 3016.214382] [ 3016.215608] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3016.218487] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3016.226794] do_raw_spin_lock+0x177/0x190 [ 3016.229142] _raw_spin_lock+0x21/0x30 [ 3016.230702] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3016.234546] ? do_raw_spin_unlock+0x75/0x190 [ 3016.241202] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3016.247175] ? do_wait_intr+0xf0/0xf0 [ 3016.249865] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3016.253239] mdc_reint+0x2d/0x110 [mdc] [ 3016.255171] mdc_unlink+0x658/0x870 [mdc] [ 3016.258484] lmv_unlink+0x34d/0xbd0 [lmv] [ 3016.261457] ll_unlink+0x24d/0x7f0 [lustre] [ 3016.267453] vfs_unlink+0x202/0x330 [ 3016.269351] do_unlinkat+0x2fa/0x460 [ 3016.272674] __x64_sys_unlink+0x37/0x50 [ 3016.275231] do_syscall_64+0xc1/0x440 [ 3016.278658] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3016.284103] RIP: 0033:0x7f83e5f5c10b [ 3016.286648] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3016.301098] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3016.309979] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3016.313841] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3016.319652] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3016.323001] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3016.324925] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3024.066406] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3024.069685] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3024.129968] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3024.137294] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3024.142166] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3024.145456] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3024.150723] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3024.161214] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3024.164114] RAX: 0000000000007184 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3024.167579] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3024.172155] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3024.176079] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3024.179760] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3024.183859] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3024.188485] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3024.191930] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3024.195458] Call Trace: [ 3024.197012] [ 3024.198483] ? show_regs.cold.9+0x22/0x2f [ 3024.200738] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3024.203689] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3024.205455] ? watchdog+0x40/0x40 [ 3024.206968] ? hrtimer_interrupt+0x128/0x2c0 [ 3024.209134] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3024.211665] ? apic_timer_interrupt+0xf/0x20 [ 3024.213849] [ 3024.215246] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3024.218447] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3024.221912] do_raw_spin_lock+0x177/0x190 [ 3024.224263] _raw_spin_lock+0x21/0x30 [ 3024.226137] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3024.229016] process_one_work+0x2c8/0x700 [ 3024.230851] worker_thread+0x296/0x6e0 [ 3024.232947] ? rescuer_thread+0x570/0x570 [ 3024.234390] kthread+0x1d1/0x200 [ 3024.236174] ? set_kthread_struct+0x70/0x70 [ 3024.238386] ret_from_fork+0x1f/0x30 [ 3028.057217] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 3028.059013] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 3028.064291] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3028.064371] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3028.064378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3028.064381] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3028.067526] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) [ 3028.086603] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3028.086617] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3028.086625] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3028.086627] RDX: 0000000000006b6b RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3028.086630] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3028.086632] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3028.086634] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3028.086637] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3028.086639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3028.086641] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3028.086648] Call Trace: [ 3028.086653] [ 3028.086657] ? show_regs.cold.9+0x22/0x2f [ 3028.086667] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3028.086672] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3028.086675] ? watchdog+0x40/0x40 [ 3028.086678] ? hrtimer_interrupt+0x128/0x2c0 [ 3028.086680] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3028.086686] ? apic_timer_interrupt+0xf/0x20 [ 3028.086689] [ 3028.086691] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3028.086696] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3028.086698] do_raw_spin_lock+0x177/0x190 [ 3028.086701] _raw_spin_lock+0x21/0x30 [ 3028.086706] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3028.090965] osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3028.091008] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3028.091014] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3028.091016] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3028.091026] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3028.091029] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3028.091032] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3028.091034] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3028.091035] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3028.091036] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3028.091038] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3028.091040] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3028.091042] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3028.091043] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3028.091049] Call Trace: [ 3028.091054] [ 3028.091059] ? show_regs.cold.9+0x22/0x2f [ 3028.091309] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3028.091316] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3028.091320] ? watchdog+0x40/0x40 [ 3028.091325] ? hrtimer_interrupt+0x128/0x2c0 [ 3028.091328] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3028.091336] ? apic_timer_interrupt+0xf/0x20 [ 3028.091340] [ 3028.091343] ? native_safe_halt+0x2e/0x50 [ 3028.091407] kvm_wait+0xcd/0xe0 [ 3028.091415] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3028.091423] do_raw_spin_lock+0x177/0x190 [ 3028.091427] _raw_spin_lock+0x21/0x30 [ 3028.091430] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3028.095021] obd_mod_free+0x46/0x140 [ptlrpc] [ 3028.095296] mdc_commit_open+0x91/0xb0 [mdc] [ 3028.095333] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3028.095665] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3028.096018] after_reply+0xe2c/0x1780 [ptlrpc] [ 3028.096243] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3028.096498] ? __next_timer_interrupt+0x160/0x160 [ 3028.096507] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3028.096755] ? do_wait_intr+0xf0/0xf0 [ 3028.096771] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3028.097008] kthread+0x1d1/0x200 [ 3028.097014] ? set_kthread_struct+0x70/0x70 [ 3028.097022] ret_from_fork+0x1f/0x30 [ 3028.241609] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3028.243024] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3028.244344] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3028.245704] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3028.247006] ? woken_wake_function+0x30/0x30 [ 3028.247942] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3028.249401] kthread+0x1d1/0x200 [ 3028.250121] ? set_kthread_struct+0x70/0x70 [ 3028.251050] ret_from_fork+0x1f/0x30 [ 3033.062693] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 519s! [ 3033.070500] Showing busy workqueues and worker pools: [ 3033.072786] workqueue events: flags=0x0 [ 3033.073873] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3033.075647] pending: osc_grant_work_handler [osc], push_to_pool [ 3033.078385] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3033.081277] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3033.084671] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3033.087503] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3033.091133] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3033.093775] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3033.098061] workqueue events_long: flags=0x0 [ 3033.100490] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3033.102921] pending: lu_objects_flush [obdclass] [ 3033.104718] workqueue events_unbound: flags=0x2 [ 3033.105769] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3033.107542] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3033.109616] workqueue events_power_efficient: flags=0x80 [ 3033.111949] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3033.114761] pending: neigh_periodic_work, do_cache_clean [ 3033.116172] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3033.118315] pending: neigh_periodic_work, check_lifetime [ 3033.120836] workqueue mm_percpu_wq: flags=0x8 [ 3033.122711] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3033.125796] pending: vmstat_update, lru_add_drain_per_cpu [ 3033.127915] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3033.129806] pending: vmstat_update [ 3033.130740] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3033.132701] pending: lru_add_drain_per_cpu BAR(48) [ 3033.134152] workqueue ipv6_addrconf: flags=0x40008 [ 3033.135590] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3033.138830] pending: addrconf_verify_work [ 3033.141035] workqueue knbd0-recv: flags=0x1a [ 3033.142825] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3033.145031] in-flight: 650:recv_work [ 3033.146408] workqueue ptlrpc_pinger: flags=0x2 [ 3033.147684] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3033.149350] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3033.151069] pool 8: cpus=0-3 flags=0x4 nice=0 hung=519s workers=4 idle: 36 466 29757 [ 3033.153219] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3023s workers=2 idle: 652 [ 3044.062071] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 3044.072927] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3044.125171] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3044.132583] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3044.138286] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3044.142668] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3044.154020] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3044.159961] RAX: 0000000000004907 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3044.164920] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3044.171084] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3044.180892] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3044.192877] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3044.202915] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3044.212136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3044.217724] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3044.222715] Call Trace: [ 3044.225956] [ 3044.227012] ? show_regs.cold.9+0x22/0x2f [ 3044.228918] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3044.233999] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3044.240860] ? watchdog+0x40/0x40 [ 3044.245670] ? hrtimer_interrupt+0x128/0x2c0 [ 3044.249969] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3044.259055] ? apic_timer_interrupt+0xf/0x20 [ 3044.261732] [ 3044.263921] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3044.269117] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3044.273153] do_raw_spin_lock+0x177/0x190 [ 3044.277360] _raw_spin_lock+0x21/0x30 [ 3044.279326] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3044.284152] ? do_raw_spin_unlock+0x75/0x190 [ 3044.289593] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3044.294437] ? do_wait_intr+0xf0/0xf0 [ 3044.296849] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3044.302995] mdc_reint+0x2d/0x110 [mdc] [ 3044.304220] mdc_unlink+0x658/0x870 [mdc] [ 3044.310482] lmv_unlink+0x34d/0xbd0 [lmv] [ 3044.311955] ll_unlink+0x24d/0x7f0 [lustre] [ 3044.319445] vfs_unlink+0x202/0x330 [ 3044.323551] do_unlinkat+0x2fa/0x460 [ 3044.325120] __x64_sys_unlink+0x37/0x50 [ 3044.329813] do_syscall_64+0xc1/0x440 [ 3044.333057] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3044.339553] RIP: 0033:0x7f83e5f5c10b [ 3044.343097] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3044.361103] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3044.371698] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3044.375559] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3044.383947] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3044.392850] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3044.400395] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3052.060760] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3052.070502] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3052.114813] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3052.120929] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3052.131847] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3052.140537] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3052.145206] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3052.167245] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3052.170659] RAX: 0000000000002497 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3052.172840] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3052.175655] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3052.178605] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3052.182384] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3052.185822] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3052.189320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3052.191665] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3052.195325] Call Trace: [ 3052.196930] [ 3052.197986] ? show_regs.cold.9+0x22/0x2f [ 3052.200240] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3052.203226] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3052.204665] ? watchdog+0x40/0x40 [ 3052.205742] ? hrtimer_interrupt+0x128/0x2c0 [ 3052.207982] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3052.212690] ? apic_timer_interrupt+0xf/0x20 [ 3052.214825] [ 3052.219585] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3052.223265] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3052.228960] do_raw_spin_lock+0x177/0x190 [ 3052.232794] _raw_spin_lock+0x21/0x30 [ 3052.237821] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3052.245842] process_one_work+0x2c8/0x700 [ 3052.250534] worker_thread+0x296/0x6e0 [ 3052.254815] ? rescuer_thread+0x570/0x570 [ 3052.260298] kthread+0x1d1/0x200 [ 3052.263069] ? set_kthread_struct+0x70/0x70 [ 3052.272718] ret_from_fork+0x1f/0x30 [ 3056.059905] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 3056.068029] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3056.128353] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3056.137742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3056.140752] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3056.142648] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3056.150207] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3056.152519] RAX: 0000000000005109 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3056.154937] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3056.156630] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3056.165654] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3056.170704] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3056.174292] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3056.181419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3056.185070] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3056.190460] Call Trace: [ 3056.192757] [ 3056.193578] ? show_regs.cold.9+0x22/0x2f [ 3056.204075] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3056.208539] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3056.215276] ? watchdog+0x40/0x40 [ 3056.216403] ? hrtimer_interrupt+0x128/0x2c0 [ 3056.224078] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3056.228425] ? apic_timer_interrupt+0xf/0x20 [ 3056.235079] [ 3056.236447] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3056.241008] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3056.246451] do_raw_spin_lock+0x177/0x190 [ 3056.248886] _raw_spin_lock+0x21/0x30 [ 3056.251323] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3056.254907] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3056.258046] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3056.261867] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3056.270876] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3056.276668] ? woken_wake_function+0x30/0x30 [ 3056.283205] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3056.290689] kthread+0x1d1/0x200 [ 3056.293936] ? set_kthread_struct+0x70/0x70 [ 3056.296299] ret_from_fork+0x1f/0x30 [ 3060.057186] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 3060.059349] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3060.095394] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3060.099613] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3060.103842] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3060.106982] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3060.115060] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3060.119462] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3060.126119] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3060.133645] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3060.137007] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3060.141656] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3060.145371] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3060.148995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3060.151836] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3060.160418] Call Trace: [ 3060.162769] [ 3060.164778] ? show_regs.cold.9+0x22/0x2f [ 3060.166437] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3060.168955] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3060.171644] ? watchdog+0x40/0x40 [ 3060.177257] ? hrtimer_interrupt+0x128/0x2c0 [ 3060.179811] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3060.185245] ? apic_timer_interrupt+0xf/0x20 [ 3060.188783] [ 3060.190461] ? native_safe_halt+0x2e/0x50 [ 3060.194049] kvm_wait+0xcd/0xe0 [ 3060.195751] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3060.199042] do_raw_spin_lock+0x177/0x190 [ 3060.203147] _raw_spin_lock+0x21/0x30 [ 3060.209193] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3060.214186] obd_mod_free+0x46/0x140 [ptlrpc] [ 3060.223249] mdc_commit_open+0x91/0xb0 [mdc] [ 3060.228051] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3060.229556] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3060.231190] after_reply+0xe2c/0x1780 [ptlrpc] [ 3060.239667] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3060.256441] ? __next_timer_interrupt+0x160/0x160 [ 3060.263448] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3060.268894] ? do_wait_intr+0xf0/0xf0 [ 3060.275444] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3060.287620] kthread+0x1d1/0x200 [ 3060.290725] ? set_kthread_struct+0x70/0x70 [ 3060.291943] ret_from_fork+0x1f/0x30 [ 3063.267662] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 550s! [ 3063.276721] Showing busy workqueues and worker pools: [ 3063.281592] workqueue events: flags=0x0 [ 3063.285333] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3063.291612] pending: osc_grant_work_handler [osc], push_to_pool [ 3063.295015] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3063.299093] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3063.302658] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3063.311528] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3063.316069] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3063.319841] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3063.329026] workqueue events_long: flags=0x0 [ 3063.331521] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3063.334819] pending: lu_objects_flush [obdclass] [ 3063.338585] workqueue events_unbound: flags=0x2 [ 3063.341377] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3063.344706] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3063.347655] workqueue events_power_efficient: flags=0x80 [ 3063.354838] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3063.361681] pending: neigh_periodic_work, do_cache_clean [ 3063.370046] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3063.374861] pending: neigh_periodic_work, check_lifetime [ 3063.378010] workqueue mm_percpu_wq: flags=0x8 [ 3063.380229] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3063.384176] pending: vmstat_update, lru_add_drain_per_cpu [ 3063.387926] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3063.390709] pending: vmstat_update [ 3063.392489] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3063.400218] pending: lru_add_drain_per_cpu BAR(48) [ 3063.403721] workqueue ipv6_addrconf: flags=0x40008 [ 3063.408659] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3063.415322] pending: addrconf_verify_work [ 3063.422112] workqueue knbd0-recv: flags=0x1a [ 3063.423647] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3063.431962] in-flight: 650:recv_work [ 3063.435911] workqueue ptlrpc_pinger: flags=0x2 [ 3063.440372] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3063.445674] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3063.452905] pool 8: cpus=0-3 flags=0x4 nice=0 hung=550s workers=4 idle: 36 466 29757 [ 3063.456138] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3053s workers=2 idle: 652 [ 3072.062550] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 3072.068338] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3072.112193] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3072.121922] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3072.128926] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3072.137185] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3072.175579] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3072.185089] RAX: 0000000000005e07 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3072.196990] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3072.206064] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3072.216503] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3072.223846] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3072.228203] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3072.244241] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3072.246760] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3072.255685] Call Trace: [ 3072.259331] [ 3072.260797] ? show_regs.cold.9+0x22/0x2f [ 3072.264318] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3072.272029] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3072.274527] ? watchdog+0x40/0x40 [ 3072.276447] ? hrtimer_interrupt+0x128/0x2c0 [ 3072.279580] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3072.282338] ? apic_timer_interrupt+0xf/0x20 [ 3072.284997] [ 3072.286314] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3072.289527] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3072.291958] do_raw_spin_lock+0x177/0x190 [ 3072.293593] _raw_spin_lock+0x21/0x30 [ 3072.295477] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3072.298304] ? do_raw_spin_unlock+0x75/0x190 [ 3072.300465] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3072.303081] ? do_wait_intr+0xf0/0xf0 [ 3072.305083] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3072.307907] mdc_reint+0x2d/0x110 [mdc] [ 3072.309511] mdc_unlink+0x658/0x870 [mdc] [ 3072.311703] lmv_unlink+0x34d/0xbd0 [lmv] [ 3072.313791] ll_unlink+0x24d/0x7f0 [lustre] [ 3072.316859] vfs_unlink+0x202/0x330 [ 3072.318894] do_unlinkat+0x2fa/0x460 [ 3072.320511] __x64_sys_unlink+0x37/0x50 [ 3072.322602] do_syscall_64+0xc1/0x440 [ 3072.323961] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3072.326427] RIP: 0033:0x7f83e5f5c10b [ 3072.328407] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3072.337922] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3072.341871] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3072.345504] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3072.349176] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3072.352695] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3072.356018] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3080.063792] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3080.069058] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3080.115007] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3080.131343] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3080.136835] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3080.142934] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3080.150652] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3080.172044] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3080.177640] RAX: 000000000000386b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3080.184621] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3080.188911] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3080.196236] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3080.201749] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3080.208750] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3080.214226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3080.217600] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3080.226235] Call Trace: [ 3080.227212] [ 3080.228423] ? show_regs.cold.9+0x22/0x2f [ 3080.234671] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3080.238806] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3080.243633] ? watchdog+0x40/0x40 [ 3080.246709] ? hrtimer_interrupt+0x128/0x2c0 [ 3080.250430] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3080.253039] ? apic_timer_interrupt+0xf/0x20 [ 3080.257672] [ 3080.258798] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3080.261467] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3080.264608] do_raw_spin_lock+0x177/0x190 [ 3080.269477] _raw_spin_lock+0x21/0x30 [ 3080.271057] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3080.275621] process_one_work+0x2c8/0x700 [ 3080.277722] worker_thread+0x296/0x6e0 [ 3080.279030] ? rescuer_thread+0x570/0x570 [ 3080.282282] kthread+0x1d1/0x200 [ 3080.284677] ? set_kthread_struct+0x70/0x70 [ 3080.288960] ret_from_fork+0x1f/0x30 [ 3084.062461] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 3084.071526] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3084.097549] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3084.103647] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3084.108178] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3084.110612] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3084.128126] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3084.130746] RAX: 0000000000007a85 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3084.136724] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3084.142747] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3084.148510] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3084.154710] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3084.159180] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3084.165876] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3084.169337] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3084.172400] Call Trace: [ 3084.174019] [ 3084.176835] ? show_regs.cold.9+0x22/0x2f [ 3084.180766] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3084.187804] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3084.192373] ? watchdog+0x40/0x40 [ 3084.198805] ? hrtimer_interrupt+0x128/0x2c0 [ 3084.200738] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3084.204905] ? apic_timer_interrupt+0xf/0x20 [ 3084.206738] [ 3084.209670] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3084.213151] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3084.219726] do_raw_spin_lock+0x177/0x190 [ 3084.221192] _raw_spin_lock+0x21/0x30 [ 3084.227954] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3084.235507] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3084.241778] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3084.246473] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3084.251320] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3084.261014] ? woken_wake_function+0x30/0x30 [ 3084.263405] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3084.273402] kthread+0x1d1/0x200 [ 3084.277346] ? set_kthread_struct+0x70/0x70 [ 3084.279602] ret_from_fork+0x1f/0x30 [ 3088.057920] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 3088.062250] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3088.095480] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3088.102938] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3088.110003] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3088.114110] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3088.143863] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3088.157500] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3088.168788] RDX: 00000000000003e1 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3088.184790] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3088.193480] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3088.199678] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3088.203642] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3088.209752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3088.216738] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3088.225061] Call Trace: [ 3088.229392] [ 3088.231526] ? show_regs.cold.9+0x22/0x2f [ 3088.239004] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3088.243746] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3088.246615] ? watchdog+0x40/0x40 [ 3088.258780] ? hrtimer_interrupt+0x128/0x2c0 [ 3088.263622] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3088.268548] ? apic_timer_interrupt+0xf/0x20 [ 3088.273526] [ 3088.274509] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3088.281637] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3088.288847] do_raw_spin_lock+0x177/0x190 [ 3088.290959] _raw_spin_lock+0x21/0x30 [ 3088.296702] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3088.298981] obd_mod_free+0x46/0x140 [ptlrpc] [ 3088.305597] mdc_commit_open+0x91/0xb0 [mdc] [ 3088.311687] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3088.318613] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3088.324565] after_reply+0xe2c/0x1780 [ptlrpc] [ 3088.327423] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3088.331540] ? __next_timer_interrupt+0x160/0x160 [ 3088.336001] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3088.338837] ? do_wait_intr+0xf0/0xf0 [ 3088.340978] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3088.344942] kthread+0x1d1/0x200 [ 3088.353230] ? set_kthread_struct+0x70/0x70 [ 3088.354564] ret_from_fork+0x1f/0x30 [ 3093.473290] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 580s! [ 3093.482033] Showing busy workqueues and worker pools: [ 3093.487214] workqueue events: flags=0x0 [ 3093.489971] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3093.494884] pending: osc_grant_work_handler [osc], push_to_pool [ 3093.499501] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3093.503375] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3093.511058] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3093.523036] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3093.531377] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3093.543896] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3093.556450] workqueue events_long: flags=0x0 [ 3093.563023] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3093.576604] pending: lu_objects_flush [obdclass] [ 3093.596648] workqueue events_unbound: flags=0x2 [ 3093.606071] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3093.618070] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3093.628125] workqueue events_power_efficient: flags=0x80 [ 3093.635134] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3093.642010] pending: neigh_periodic_work, do_cache_clean [ 3093.655643] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3093.668020] pending: neigh_periodic_work, check_lifetime [ 3093.678274] workqueue mm_percpu_wq: flags=0x8 [ 3093.683199] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3093.692072] pending: vmstat_update, lru_add_drain_per_cpu [ 3093.701717] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3093.708470] pending: vmstat_update [ 3093.714107] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3093.719538] pending: lru_add_drain_per_cpu BAR(48) [ 3093.730937] workqueue ipv6_addrconf: flags=0x40008 [ 3093.742485] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3093.755230] pending: addrconf_verify_work [ 3093.766038] workqueue knbd0-recv: flags=0x1a [ 3093.767168] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3093.778715] in-flight: 650:recv_work [ 3093.786806] workqueue ptlrpc_pinger: flags=0x2 [ 3093.801021] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3093.818963] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3093.831027] pool 8: cpus=0-3 flags=0x4 nice=0 hung=580s workers=4 idle: 36 466 29757 [ 3093.845404] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3083s workers=2 idle: 652 [ 3100.062641] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 3100.068538] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3100.111048] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3100.120326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3100.127479] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3100.131592] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3100.145669] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3100.153632] RAX: 0000000000007474 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3100.160187] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3100.171303] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3100.184304] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3100.191467] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3100.197174] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3100.200995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3100.203992] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3100.213200] Call Trace: [ 3100.216475] [ 3100.220168] ? show_regs.cold.9+0x22/0x2f [ 3100.228156] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3100.237680] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3100.243355] ? watchdog+0x40/0x40 [ 3100.249556] ? hrtimer_interrupt+0x128/0x2c0 [ 3100.252657] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3100.255195] ? apic_timer_interrupt+0xf/0x20 [ 3100.257178] [ 3100.257725] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3100.262399] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3100.264039] do_raw_spin_lock+0x177/0x190 [ 3100.267119] _raw_spin_lock+0x21/0x30 [ 3100.268066] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3100.269401] ? do_raw_spin_unlock+0x75/0x190 [ 3100.270437] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3100.273601] ? do_wait_intr+0xf0/0xf0 [ 3100.277627] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3100.284210] mdc_reint+0x2d/0x110 [mdc] [ 3100.288573] mdc_unlink+0x658/0x870 [mdc] [ 3100.294469] lmv_unlink+0x34d/0xbd0 [lmv] [ 3100.300048] ll_unlink+0x24d/0x7f0 [lustre] [ 3100.305604] vfs_unlink+0x202/0x330 [ 3100.312918] do_unlinkat+0x2fa/0x460 [ 3100.315687] __x64_sys_unlink+0x37/0x50 [ 3100.319241] do_syscall_64+0xc1/0x440 [ 3100.322560] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3100.327568] RIP: 0033:0x7f83e5f5c10b [ 3100.336121] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3100.364072] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3100.371414] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3100.380148] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3100.386857] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3100.400973] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3100.407482] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3108.064328] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3108.070680] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3108.104351] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3108.115062] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3108.123831] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3108.129686] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3108.138285] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3108.153573] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3108.158283] RAX: 0000000000007e84 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3108.163612] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3108.167902] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3108.174127] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3108.182613] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3108.192487] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3108.198167] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3108.205836] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3108.208111] Call Trace: [ 3108.213116] [ 3108.213797] ? show_regs.cold.9+0x22/0x2f [ 3108.216590] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3108.221084] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3108.226974] ? watchdog+0x40/0x40 [ 3108.228542] ? hrtimer_interrupt+0x128/0x2c0 [ 3108.232952] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3108.236853] ? apic_timer_interrupt+0xf/0x20 [ 3108.243140] [ 3108.245286] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3108.250189] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3108.254933] do_raw_spin_lock+0x177/0x190 [ 3108.258500] _raw_spin_lock+0x21/0x30 [ 3108.260272] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3108.265527] process_one_work+0x2c8/0x700 [ 3108.270124] worker_thread+0x296/0x6e0 [ 3108.273618] ? rescuer_thread+0x570/0x570 [ 3108.279237] kthread+0x1d1/0x200 [ 3108.281053] ? set_kthread_struct+0x70/0x70 [ 3108.283060] ret_from_fork+0x1f/0x30 [ 3112.059936] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 3112.083183] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3112.165668] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3112.175523] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3112.193928] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3112.198320] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3112.219872] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3112.233589] RAX: 0000000000007b89 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3112.245534] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3112.249545] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3112.260625] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3112.274826] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3112.285173] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3112.299586] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3112.301972] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3112.310388] Call Trace: [ 3112.313451] [ 3112.320362] ? show_regs.cold.9+0x22/0x2f [ 3112.325370] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3112.330643] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3112.334691] ? watchdog+0x40/0x40 [ 3112.339152] ? hrtimer_interrupt+0x128/0x2c0 [ 3112.340687] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3112.354787] ? apic_timer_interrupt+0xf/0x20 [ 3112.360275] [ 3112.364012] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3112.373406] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3112.382729] do_raw_spin_lock+0x177/0x190 [ 3112.384273] _raw_spin_lock+0x21/0x30 [ 3112.387171] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3112.404484] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3112.406614] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3112.417355] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3112.428460] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3112.430673] ? woken_wake_function+0x30/0x30 [ 3112.443275] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3112.454293] kthread+0x1d1/0x200 [ 3112.455542] ? set_kthread_struct+0x70/0x70 [ 3112.467919] ret_from_fork+0x1f/0x30 [ 3120.056086] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 3120.061520] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3120.087501] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3120.092445] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3120.096138] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3120.098491] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3120.105392] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3120.107391] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3120.109866] RDX: 00000000000058e9 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3120.112241] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3120.114554] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3120.117654] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3120.120601] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3120.124170] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3120.126971] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3120.130760] Call Trace: [ 3120.131794] [ 3120.132524] ? show_regs.cold.9+0x22/0x2f [ 3120.134088] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3120.136080] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3120.138165] ? watchdog+0x40/0x40 [ 3120.139565] ? hrtimer_interrupt+0x128/0x2c0 [ 3120.141404] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3120.143370] ? apic_timer_interrupt+0xf/0x20 [ 3120.145258] [ 3120.146078] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3120.148297] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3120.150146] do_raw_spin_lock+0x177/0x190 [ 3120.152030] _raw_spin_lock+0x21/0x30 [ 3120.153430] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3120.155262] obd_mod_free+0x46/0x140 [ptlrpc] [ 3120.157366] mdc_commit_open+0x91/0xb0 [mdc] [ 3120.159223] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3120.161441] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3120.164049] after_reply+0xe2c/0x1780 [ptlrpc] [ 3120.166623] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3120.169311] ? __next_timer_interrupt+0x160/0x160 [ 3120.171476] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3120.173434] ? do_wait_intr+0xf0/0xf0 [ 3120.175197] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3120.177791] kthread+0x1d1/0x200 [ 3120.178973] ? set_kthread_struct+0x70/0x70 [ 3120.180810] ret_from_fork+0x1f/0x30 [ 3120.979473] rcu: INFO: rcu_sched self-detected stall on CPU [ 3120.981419] rcu: 0-....: (738577 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 3120.990738] (t=1140031 jiffies g=262441 q=57295) [ 3120.993895] rcu: rcu_sched kthread starved for 609221 jiffies! g262441 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 3120.998335] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3121.004279] rcu: RCU grace-period kthread stack dump: [ 3121.009133] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3121.021827] Call Trace: [ 3121.023449] __schedule+0x351/0xcb0 [ 3121.025240] schedule+0xc0/0x180 [ 3121.026335] schedule_timeout+0xb4/0x190 [ 3121.029233] ? do_raw_spin_unlock+0x75/0x190 [ 3121.035172] ? __next_timer_interrupt+0x160/0x160 [ 3121.037825] ? prepare_to_swait_event+0xf3/0x140 [ 3121.042514] rcu_gp_kthread+0x8dd/0x15b0 [ 3121.048322] ? force_qs_rnp+0x2c0/0x2c0 [ 3121.052712] kthread+0x1d1/0x200 [ 3121.055178] ? set_kthread_struct+0x70/0x70 [ 3121.059290] ret_from_fork+0x1f/0x30 [ 3121.068367] rcu: Stack dump where RCU GP kthread last ran: [ 3121.072747] Sending NMI from CPU 0 to CPUs 2: [ 3121.079301] NMI backtrace for cpu 2 [ 3121.079309] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3121.079311] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3121.079312] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3121.079315] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3121.079316] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3121.079318] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 [ 3121.079320] RAX: 000000000000728a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3121.079321] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3121.079322] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3121.079324] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3121.079325] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3121.079326] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3121.079327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3121.079328] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3121.079330] Call Trace: [ 3121.079330] [ 3121.079331] ? show_regs.cold.9+0x22/0x2f [ 3121.079332] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3121.079333] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3121.079334] ? nmi_handle+0xbd/0x210 [ 3121.079335] ? default_do_nmi+0x4b/0x1b0 [ 3121.079336] ? do_nmi+0x280/0x380 [ 3121.079337] ? end_repeat_nmi+0x16/0x1f [ 3121.079338] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3121.079339] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3121.079340] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3121.079341] [ 3121.079342] do_raw_spin_lock+0x177/0x190 [ 3121.079343] _raw_spin_lock+0x21/0x30 [ 3121.079344] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3121.079345] process_one_work+0x2c8/0x700 [ 3121.079346] worker_thread+0x296/0x6e0 [ 3121.079347] ? rescuer_thread+0x570/0x570 [ 3121.079348] kthread+0x1d1/0x200 [ 3121.079349] ? set_kthread_struct+0x70/0x70 [ 3121.079350] ret_from_fork+0x1f/0x30 [ 3121.085317] NMI backtrace for cpu 0 [ 3121.324806] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3121.334915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3121.345125] Call Trace: [ 3121.348117] [ 3121.350045] ? dump_stack+0xbb/0x10e [ 3121.352760] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3121.356762] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3121.358797] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3121.362308] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3121.364529] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3121.366612] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3121.372690] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3121.375311] ? update_process_times+0x6d/0xc0 [ 3121.377389] ? tick_sched_handle.isra.18+0x27/0x90 [ 3121.385842] ? tick_sched_timer+0x53/0xb0 [ 3121.389790] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3121.391765] ? tick_sched_handle.isra.18+0x90/0x90 [ 3121.397723] ? hrtimer_interrupt+0x128/0x2c0 [ 3121.399649] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3121.403291] ? apic_timer_interrupt+0xf/0x20 [ 3121.404619] [ 3121.408533] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3121.413209] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3121.416275] ? do_raw_spin_lock+0x177/0x190 [ 3121.419920] ? _raw_spin_lock+0x21/0x30 [ 3121.422923] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3121.430753] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3121.435567] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3121.438340] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3121.443558] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3121.446494] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3121.448744] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3121.459239] ? __next_timer_interrupt+0x160/0x160 [ 3121.460823] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3121.463386] ? do_wait_intr+0xf0/0xf0 [ 3121.466543] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3121.469809] ? kthread+0x1d1/0x200 [ 3121.471283] ? set_kthread_struct+0x70/0x70 [ 3121.475397] ? ret_from_fork+0x1f/0x30 [ 3124.193609] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 611s! [ 3124.198991] Showing busy workqueues and worker pools: [ 3124.203512] workqueue events: flags=0x0 [ 3124.207676] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3124.212321] pending: osc_grant_work_handler [osc], push_to_pool [ 3124.217264] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3124.221629] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3124.225553] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3124.229933] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3124.233277] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3124.236608] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3124.240316] workqueue events_long: flags=0x0 [ 3124.242442] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3124.247237] pending: lu_objects_flush [obdclass] [ 3124.250513] workqueue events_unbound: flags=0x2 [ 3124.252668] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3124.255627] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3124.258456] workqueue events_power_efficient: flags=0x80 [ 3124.263333] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3124.271784] pending: neigh_periodic_work, do_cache_clean [ 3124.277198] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3124.280521] pending: neigh_periodic_work, check_lifetime [ 3124.288030] workqueue mm_percpu_wq: flags=0x8 [ 3124.292386] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3124.300970] pending: vmstat_update, lru_add_drain_per_cpu [ 3124.307618] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3124.316354] pending: vmstat_update [ 3124.319766] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3124.328105] pending: lru_add_drain_per_cpu BAR(48) [ 3124.335019] workqueue ipv6_addrconf: flags=0x40008 [ 3124.336946] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3124.340899] pending: addrconf_verify_work [ 3124.347045] workqueue knbd0-recv: flags=0x1a [ 3124.355495] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3124.363733] in-flight: 650:recv_work [ 3124.369022] workqueue ptlrpc_pinger: flags=0x2 [ 3124.372250] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3124.379603] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3124.385364] pool 8: cpus=0-3 flags=0x4 nice=0 hung=611s workers=4 idle: 36 466 29757 [ 3124.394392] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3114s workers=2 idle: 652 [ 3128.062218] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 3128.067935] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3128.091618] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3128.097047] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3128.100208] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3128.101680] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3128.108048] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3128.110494] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3128.112677] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3128.114721] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3128.117124] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3128.119284] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3128.122103] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3128.124604] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3128.126446] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3128.128624] Call Trace: [ 3128.129407] [ 3128.130103] ? show_regs.cold.9+0x22/0x2f [ 3128.131386] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3128.133295] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3128.134434] ? watchdog+0x40/0x40 [ 3128.135356] ? hrtimer_interrupt+0x128/0x2c0 [ 3128.136659] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3128.138226] ? apic_timer_interrupt+0xf/0x20 [ 3128.147646] [ 3128.158632] ? native_safe_halt+0x2e/0x50 [ 3128.162413] kvm_wait+0xcd/0xe0 [ 3128.165456] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3128.166770] do_raw_spin_lock+0x177/0x190 [ 3128.171006] _raw_spin_lock+0x21/0x30 [ 3128.174405] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3128.177240] ? do_raw_spin_unlock+0x75/0x190 [ 3128.180200] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3128.184681] ? do_wait_intr+0xf0/0xf0 [ 3128.186035] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3128.193227] mdc_reint+0x2d/0x110 [mdc] [ 3128.197668] mdc_unlink+0x658/0x870 [mdc] [ 3128.201461] lmv_unlink+0x34d/0xbd0 [lmv] [ 3128.206458] ll_unlink+0x24d/0x7f0 [lustre] [ 3128.208440] vfs_unlink+0x202/0x330 [ 3128.210062] do_unlinkat+0x2fa/0x460 [ 3128.213765] __x64_sys_unlink+0x37/0x50 [ 3128.215437] do_syscall_64+0xc1/0x440 [ 3128.217200] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3128.219210] RIP: 0033:0x7f83e5f5c10b [ 3128.220693] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3128.233472] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3128.242403] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3128.248696] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3128.253020] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3128.256588] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3128.259484] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3140.059387] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 3140.066314] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3140.114682] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3140.129825] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3140.137389] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3140.139436] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3140.148896] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3140.151838] RAX: 0000000000007e0b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3140.154051] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3140.159865] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3140.163267] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3140.166228] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3140.172517] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3140.176895] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3140.179753] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3140.183234] Call Trace: [ 3140.184656] [ 3140.185362] ? show_regs.cold.9+0x22/0x2f [ 3140.186772] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3140.190187] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3140.192940] ? watchdog+0x40/0x40 [ 3140.195853] ? hrtimer_interrupt+0x128/0x2c0 [ 3140.198272] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3140.202866] ? apic_timer_interrupt+0xf/0x20 [ 3140.213013] [ 3140.217394] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3140.224543] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3140.228183] do_raw_spin_lock+0x177/0x190 [ 3140.232576] _raw_spin_lock+0x21/0x30 [ 3140.235991] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3140.242528] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3140.245516] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3140.250357] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3140.258938] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3140.268578] ? woken_wake_function+0x30/0x30 [ 3140.270069] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3140.277180] kthread+0x1d1/0x200 [ 3140.284118] ? set_kthread_struct+0x70/0x70 [ 3140.286788] ret_from_fork+0x1f/0x30 [ 3148.060141] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3148.065980] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3148.092514] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3148.099037] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3148.103524] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3148.106641] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3148.109157] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3148.117595] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3148.121145] RAX: 0000000000004809 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3148.124705] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3148.128028] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3148.131510] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3148.134983] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3148.138845] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3148.145724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3148.148274] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3148.151989] Call Trace: [ 3148.153192] [ 3148.153978] ? show_regs.cold.9+0x22/0x2f [ 3148.155757] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3148.157483] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3148.159145] ? watchdog+0x40/0x40 [ 3148.162142] ? hrtimer_interrupt+0x128/0x2c0 [ 3148.164033] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3148.166069] ? apic_timer_interrupt+0xf/0x20 [ 3148.168076] [ 3148.168884] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3148.171381] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3148.173925] do_raw_spin_lock+0x177/0x190 [ 3148.175898] _raw_spin_lock+0x21/0x30 [ 3148.177441] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3148.179823] process_one_work+0x2c8/0x700 [ 3148.181439] worker_thread+0x296/0x6e0 [ 3148.183021] ? rescuer_thread+0x570/0x570 [ 3148.184797] kthread+0x1d1/0x200 [ 3148.186429] ? set_kthread_struct+0x70/0x70 [ 3148.188029] ret_from_fork+0x1f/0x30 [ 3152.056363] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 3152.060664] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3152.086814] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3152.092249] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3152.098383] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3152.102566] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3152.126843] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3152.141381] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3152.146418] RDX: 0000000000007e8a RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3152.151061] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3152.161937] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3152.167912] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3152.175433] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3152.185139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3152.196057] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3152.200990] Call Trace: [ 3152.202587] [ 3152.203983] ? show_regs.cold.9+0x22/0x2f [ 3152.207060] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3152.210691] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3152.218748] ? watchdog+0x40/0x40 [ 3152.221371] ? hrtimer_interrupt+0x128/0x2c0 [ 3152.223869] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3152.231872] ? apic_timer_interrupt+0xf/0x20 [ 3152.233114] [ 3152.233898] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3152.240220] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3152.245038] do_raw_spin_lock+0x177/0x190 [ 3152.247868] _raw_spin_lock+0x21/0x30 [ 3152.249334] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3152.252689] obd_mod_free+0x46/0x140 [ptlrpc] [ 3152.254743] mdc_commit_open+0x91/0xb0 [mdc] [ 3152.256416] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3152.258815] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3152.261727] after_reply+0xe2c/0x1780 [ptlrpc] [ 3152.264810] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3152.269415] ? __next_timer_interrupt+0x160/0x160 [ 3152.276140] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3152.279457] ? do_wait_intr+0xf0/0xf0 [ 3152.286693] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3152.288430] kthread+0x1d1/0x200 [ 3152.290088] ? set_kthread_struct+0x70/0x70 [ 3152.292217] ret_from_fork+0x1f/0x30 [ 3154.922684] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 641s! [ 3154.932396] Showing busy workqueues and worker pools: [ 3154.944196] workqueue events: flags=0x0 [ 3154.949515] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3154.959915] pending: osc_grant_work_handler [osc], push_to_pool [ 3154.969382] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3154.980512] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3154.990392] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3154.995213] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3155.008421] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3155.020412] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3155.027908] workqueue events_long: flags=0x0 [ 3155.031263] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3155.034719] pending: lu_objects_flush [obdclass] [ 3155.038984] workqueue events_unbound: flags=0x2 [ 3155.040303] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3155.049259] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3155.054479] workqueue events_power_efficient: flags=0x80 [ 3155.057108] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3155.061688] pending: neigh_periodic_work, do_cache_clean [ 3155.068249] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3155.075893] pending: neigh_periodic_work, check_lifetime [ 3155.086890] workqueue mm_percpu_wq: flags=0x8 [ 3155.089328] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3155.098941] pending: vmstat_update, lru_add_drain_per_cpu [ 3155.101025] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3155.107703] pending: vmstat_update [ 3155.112356] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3155.114592] pending: lru_add_drain_per_cpu BAR(48) [ 3155.124297] workqueue ipv6_addrconf: flags=0x40008 [ 3155.134489] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3155.139587] pending: addrconf_verify_work [ 3155.141254] workqueue knbd0-recv: flags=0x1a [ 3155.151502] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3155.165899] in-flight: 650:recv_work [ 3155.167562] workqueue ptlrpc_pinger: flags=0x2 [ 3155.173395] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3155.185147] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3155.187121] pool 8: cpus=0-3 flags=0x4 nice=0 hung=642s workers=4 idle: 36 466 29757 [ 3155.193668] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3145s workers=2 idle: 652 [ 3156.062051] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 3156.065896] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3156.091493] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3156.096058] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3156.099132] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3156.101363] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3156.110563] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3156.114587] RAX: 0000000000003764 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3156.118165] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3156.121425] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3156.124098] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3156.127693] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3156.130812] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3156.134448] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3156.137479] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3156.141385] Call Trace: [ 3156.142801] [ 3156.143801] ? show_regs.cold.9+0x22/0x2f [ 3156.146112] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3156.148556] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3156.151090] ? watchdog+0x40/0x40 [ 3156.153113] ? hrtimer_interrupt+0x128/0x2c0 [ 3156.155248] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3156.157424] ? apic_timer_interrupt+0xf/0x20 [ 3156.159997] [ 3156.161083] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3156.163876] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3156.166974] do_raw_spin_lock+0x177/0x190 [ 3156.168796] _raw_spin_lock+0x21/0x30 [ 3156.170929] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3156.174546] ? do_raw_spin_unlock+0x75/0x190 [ 3156.176620] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3156.178591] ? do_wait_intr+0xf0/0xf0 [ 3156.180316] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3156.183441] mdc_reint+0x2d/0x110 [mdc] [ 3156.185332] mdc_unlink+0x658/0x870 [mdc] [ 3156.188477] lmv_unlink+0x34d/0xbd0 [lmv] [ 3156.191511] ll_unlink+0x24d/0x7f0 [lustre] [ 3156.194379] vfs_unlink+0x202/0x330 [ 3156.196327] do_unlinkat+0x2fa/0x460 [ 3156.198504] __x64_sys_unlink+0x37/0x50 [ 3156.201075] do_syscall_64+0xc1/0x440 [ 3156.203213] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3156.206749] RIP: 0033:0x7f83e5f5c10b [ 3156.207952] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3156.218461] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3156.225058] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3156.229867] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3156.234919] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3156.239775] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3156.244846] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3168.059267] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 3168.067137] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3168.142251] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3168.159109] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3168.172317] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3168.179372] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3168.189085] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3168.192924] RAX: 0000000000006117 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3168.196743] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3168.199858] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3168.203055] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3168.206884] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3168.209811] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3168.212609] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3168.215086] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3168.218794] Call Trace: [ 3168.219960] [ 3168.221125] ? show_regs.cold.9+0x22/0x2f [ 3168.224183] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3168.226532] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3168.228618] ? watchdog+0x40/0x40 [ 3168.230056] ? hrtimer_interrupt+0x128/0x2c0 [ 3168.232896] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3168.235173] ? apic_timer_interrupt+0xf/0x20 [ 3168.237171] [ 3168.238142] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3168.240941] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3168.243620] do_raw_spin_lock+0x177/0x190 [ 3168.248095] _raw_spin_lock+0x21/0x30 [ 3168.249105] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3168.251167] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3168.256008] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3168.259914] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3168.262942] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3168.271613] ? woken_wake_function+0x30/0x30 [ 3168.274186] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3168.279227] kthread+0x1d1/0x200 [ 3168.284088] ? set_kthread_struct+0x70/0x70 [ 3168.287163] ret_from_fork+0x1f/0x30 [ 3176.061323] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3176.065284] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3176.091524] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3176.097419] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3176.102718] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3176.107181] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3176.110894] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3176.123227] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3176.126273] RAX: 0000000000006c87 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3176.130271] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3176.134650] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3176.138767] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3176.142445] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3176.144989] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3176.148475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3176.150327] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3176.152754] Call Trace: [ 3176.154840] [ 3176.155781] ? show_regs.cold.9+0x22/0x2f [ 3176.157868] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3176.166693] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3176.173645] ? watchdog+0x40/0x40 [ 3176.174942] ? hrtimer_interrupt+0x128/0x2c0 [ 3176.177608] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3176.179763] ? apic_timer_interrupt+0xf/0x20 [ 3176.182001] [ 3176.183454] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3176.186226] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3176.188956] do_raw_spin_lock+0x177/0x190 [ 3176.191263] _raw_spin_lock+0x21/0x30 [ 3176.193295] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3176.196138] process_one_work+0x2c8/0x700 [ 3176.197873] worker_thread+0x296/0x6e0 [ 3176.199486] ? rescuer_thread+0x570/0x570 [ 3176.201495] kthread+0x1d1/0x200 [ 3176.202898] ? set_kthread_struct+0x70/0x70 [ 3176.205216] ret_from_fork+0x1f/0x30 [ 3180.056020] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 3180.058638] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3180.094899] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3180.105238] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3180.108724] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3180.114180] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3180.126137] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3180.133256] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3180.142655] RDX: 0000000000003e05 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3180.149976] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3180.158657] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3180.162475] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3180.165571] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3180.171343] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3180.175622] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3180.181823] Call Trace: [ 3180.182633] [ 3180.183943] ? show_regs.cold.9+0x22/0x2f [ 3180.186034] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3180.188290] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3180.190561] ? watchdog+0x40/0x40 [ 3180.194883] ? hrtimer_interrupt+0x128/0x2c0 [ 3180.200791] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3180.206443] ? apic_timer_interrupt+0xf/0x20 [ 3180.210292] [ 3180.212227] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3180.214281] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3180.218557] do_raw_spin_lock+0x177/0x190 [ 3180.220457] _raw_spin_lock+0x21/0x30 [ 3180.222818] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3180.225556] obd_mod_free+0x46/0x140 [ptlrpc] [ 3180.235922] mdc_commit_open+0x91/0xb0 [mdc] [ 3180.240382] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3180.247400] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3180.254996] after_reply+0xe2c/0x1780 [ptlrpc] [ 3180.260143] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3180.264720] ? __next_timer_interrupt+0x160/0x160 [ 3180.271136] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3180.276122] ? do_wait_intr+0xf0/0xf0 [ 3180.279463] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3180.285526] kthread+0x1d1/0x200 [ 3180.290347] ? set_kthread_struct+0x70/0x70 [ 3180.293881] ret_from_fork+0x1f/0x30 [ 3184.062634] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 3184.066685] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3184.098839] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3184.106498] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3184.110717] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3184.113521] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3184.128473] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3184.131843] RAX: 0000000000007286 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3184.135552] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3184.142495] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3184.150965] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3184.160803] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3184.168097] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3184.175372] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3184.180968] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3184.185544] Call Trace: [ 3184.186732] [ 3184.188051] ? show_regs.cold.9+0x22/0x2f [ 3184.197098] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3184.203216] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3184.205753] ? watchdog+0x40/0x40 [ 3184.208408] ? hrtimer_interrupt+0x128/0x2c0 [ 3184.212467] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3184.220647] ? apic_timer_interrupt+0xf/0x20 [ 3184.225507] [ 3184.226649] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3184.230675] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3184.233569] do_raw_spin_lock+0x177/0x190 [ 3184.236640] _raw_spin_lock+0x21/0x30 [ 3184.241709] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3184.245729] ? do_raw_spin_unlock+0x75/0x190 [ 3184.248543] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3184.253746] ? do_wait_intr+0xf0/0xf0 [ 3184.255766] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3184.262576] mdc_reint+0x2d/0x110 [mdc] [ 3184.264383] mdc_unlink+0x658/0x870 [mdc] [ 3184.267633] lmv_unlink+0x34d/0xbd0 [lmv] [ 3184.269699] ll_unlink+0x24d/0x7f0 [lustre] [ 3184.274007] vfs_unlink+0x202/0x330 [ 3184.275750] do_unlinkat+0x2fa/0x460 [ 3184.278610] __x64_sys_unlink+0x37/0x50 [ 3184.281711] do_syscall_64+0xc1/0x440 [ 3184.286506] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3184.293542] RIP: 0033:0x7f83e5f5c10b [ 3184.300296] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3184.313618] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3184.317113] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3184.320382] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3184.323941] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3184.326785] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3184.333501] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3185.632688] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 672s! [ 3185.636732] Showing busy workqueues and worker pools: [ 3185.641839] workqueue events: flags=0x0 [ 3185.643605] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3185.653957] pending: osc_grant_work_handler [osc], push_to_pool [ 3185.660509] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3185.667266] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3185.675375] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3185.677983] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3185.688538] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3185.696434] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3185.710528] workqueue events_long: flags=0x0 [ 3185.714585] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3185.720018] pending: lu_objects_flush [obdclass] [ 3185.722533] workqueue events_unbound: flags=0x2 [ 3185.725836] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3185.729804] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3185.733137] workqueue events_power_efficient: flags=0x80 [ 3185.736349] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3185.740831] pending: neigh_periodic_work, do_cache_clean [ 3185.743659] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3185.746791] pending: neigh_periodic_work, check_lifetime [ 3185.749925] workqueue mm_percpu_wq: flags=0x8 [ 3185.752031] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3185.755998] pending: vmstat_update, lru_add_drain_per_cpu [ 3185.759864] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3185.763874] pending: vmstat_update [ 3185.765606] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3185.769434] pending: lru_add_drain_per_cpu BAR(48) [ 3185.772176] workqueue ipv6_addrconf: flags=0x40008 [ 3185.776513] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3185.780630] pending: addrconf_verify_work [ 3185.782243] workqueue knbd0-recv: flags=0x1a [ 3185.785504] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3185.788212] in-flight: 650:recv_work [ 3185.789928] workqueue ptlrpc_pinger: flags=0x2 [ 3185.791785] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3185.794184] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3185.797912] pool 8: cpus=0-3 flags=0x4 nice=0 hung=672s workers=4 idle: 36 466 29757 [ 3185.801901] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3175s workers=2 idle: 652 [ 3196.059036] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 3196.066838] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3196.163548] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3196.188804] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3196.208640] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3196.218749] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3196.245029] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3196.251066] RAX: 0000000000001889 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3196.266750] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3196.275651] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3196.283584] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3196.290686] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3196.298251] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3196.304713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3196.311313] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3196.319128] Call Trace: [ 3196.322381] [ 3196.323687] ? show_regs.cold.9+0x22/0x2f [ 3196.326821] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3196.331720] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3196.336780] ? watchdog+0x40/0x40 [ 3196.340634] ? hrtimer_interrupt+0x128/0x2c0 [ 3196.341980] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3196.345584] ? apic_timer_interrupt+0xf/0x20 [ 3196.350323] [ 3196.352497] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3196.357547] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3196.361925] do_raw_spin_lock+0x177/0x190 [ 3196.365795] _raw_spin_lock+0x21/0x30 [ 3196.370135] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3196.376737] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3196.384541] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3196.393899] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3196.399317] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3196.404508] ? woken_wake_function+0x30/0x30 [ 3196.407776] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3196.413718] kthread+0x1d1/0x200 [ 3196.414681] ? set_kthread_struct+0x70/0x70 [ 3196.416628] ret_from_fork+0x1f/0x30 [ 3204.061535] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 3204.074728] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3204.151014] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3204.159219] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3204.175059] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3204.187997] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3204.190843] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3204.223644] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3204.226414] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3204.239359] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3204.251917] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3204.264278] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3204.277344] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3204.284795] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3204.301759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3204.305281] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3204.314342] Call Trace: [ 3204.316409] [ 3204.317777] ? show_regs.cold.9+0x22/0x2f [ 3204.328335] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3204.333905] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3204.339740] ? watchdog+0x40/0x40 [ 3204.343286] ? hrtimer_interrupt+0x128/0x2c0 [ 3204.344449] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3204.352332] ? apic_timer_interrupt+0xf/0x20 [ 3204.355620] [ 3204.357765] ? native_safe_halt+0x2e/0x50 [ 3204.366372] kvm_wait+0xcd/0xe0 [ 3204.368926] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3204.376435] do_raw_spin_lock+0x177/0x190 [ 3204.382655] _raw_spin_lock+0x21/0x30 [ 3204.386770] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3204.393708] process_one_work+0x2c8/0x700 [ 3204.398846] worker_thread+0x296/0x6e0 [ 3204.405452] ? rescuer_thread+0x570/0x570 [ 3204.415227] kthread+0x1d1/0x200 [ 3204.423030] ? set_kthread_struct+0x70/0x70 [ 3204.430432] ret_from_fork+0x1f/0x30 [ 3212.056161] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 3212.061887] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3212.062021] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 3212.103324] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3212.107663] Modules linked in: lustre(O) [ 3212.116501] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3212.116517] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3212.116540] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3212.116546] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3212.116624] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3212.116627] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3212.116629] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3212.116632] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3212.116634] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3212.116638] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3212.116640] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3212.116643] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3212.116652] Call Trace: [ 3212.116661] [ 3212.119231] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 [ 3212.124391] ? show_regs.cold.9+0x22/0x2f [ 3212.128276] squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3212.128306] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3212.128314] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3212.128317] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3212.128331] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3212.128341] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3212.128346] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3212.128349] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3212.128351] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3212.128353] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3212.128355] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3212.128358] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3212.128361] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3212.128368] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3212.128375] Call Trace: [ 3212.128380] [ 3212.128387] ? show_regs.cold.9+0x22/0x2f [ 3212.128397] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3212.128403] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3212.128408] ? watchdog+0x40/0x40 [ 3212.128413] ? hrtimer_interrupt+0x128/0x2c0 [ 3212.128417] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3212.128424] ? apic_timer_interrupt+0xf/0x20 [ 3212.128428] [ 3212.128431] ? native_safe_halt+0x2e/0x50 [ 3212.128435] kvm_wait+0xcd/0xe0 [ 3212.128442] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3212.128448] do_raw_spin_lock+0x177/0x190 [ 3212.128453] _raw_spin_lock+0x21/0x30 [ 3212.128458] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3212.128741] ? do_raw_spin_unlock+0x75/0x190 [ 3212.128747] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3212.128944] ? do_wait_intr+0xf0/0xf0 [ 3212.128951] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3212.129134] mdc_reint+0x2d/0x110 [mdc] [ 3212.129167] mdc_unlink+0x658/0x870 [mdc] [ 3212.129192] lmv_unlink+0x34d/0xbd0 [lmv] [ 3212.129222] ll_unlink+0x24d/0x7f0 [lustre] [ 3212.129341] vfs_unlink+0x202/0x330 [ 3212.129351] do_unlinkat+0x2fa/0x460 [ 3212.129356] __x64_sys_unlink+0x37/0x50 [ 3212.129359] do_syscall_64+0xc1/0x440 [ 3212.129366] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3212.129371] RIP: 0033:0x7f83e5f5c10b [ 3212.129377] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3212.129380] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3212.129384] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3212.129386] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3212.129388] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3212.129390] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3212.129392] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3212.484859] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3212.487603] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3212.490657] ? watchdog+0x40/0x40 [ 3212.491776] ? hrtimer_interrupt+0x128/0x2c0 [ 3212.493811] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3212.497956] ? apic_timer_interrupt+0xf/0x20 [ 3212.501647] [ 3212.502873] ? native_safe_halt+0x2e/0x50 [ 3212.506166] kvm_wait+0xcd/0xe0 [ 3212.508617] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3212.514394] do_raw_spin_lock+0x177/0x190 [ 3212.517616] _raw_spin_lock+0x21/0x30 [ 3212.518852] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3212.525127] obd_mod_free+0x46/0x140 [ptlrpc] [ 3212.530059] mdc_commit_open+0x91/0xb0 [mdc] [ 3212.532557] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3212.536918] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3212.544030] after_reply+0xe2c/0x1780 [ptlrpc] [ 3212.549738] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3212.556354] ? __next_timer_interrupt+0x160/0x160 [ 3212.562699] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3212.571784] ? do_wait_intr+0xf0/0xf0 [ 3212.587330] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3212.600646] kthread+0x1d1/0x200 [ 3212.609472] ? set_kthread_struct+0x70/0x70 [ 3212.615451] ret_from_fork+0x1f/0x30 [ 3215.843196] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 702s! [ 3215.851665] Showing busy workqueues and worker pools: [ 3215.856920] workqueue events: flags=0x0 [ 3215.862270] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3215.869060] pending: osc_grant_work_handler [osc], push_to_pool [ 3215.873121] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3215.876290] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3215.880212] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3215.883614] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3215.889445] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3215.892829] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3215.896712] workqueue events_long: flags=0x0 [ 3215.899194] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3215.906230] pending: lu_objects_flush [obdclass] [ 3215.908830] workqueue events_unbound: flags=0x2 [ 3215.910427] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3215.918510] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3215.924184] workqueue events_power_efficient: flags=0x80 [ 3215.926736] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3215.935186] pending: neigh_periodic_work, do_cache_clean [ 3215.937568] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3215.940691] pending: neigh_periodic_work, check_lifetime [ 3215.944276] workqueue mm_percpu_wq: flags=0x8 [ 3215.946233] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3215.950909] pending: vmstat_update, lru_add_drain_per_cpu [ 3215.955317] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3215.959840] pending: vmstat_update [ 3215.961661] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3215.965928] pending: lru_add_drain_per_cpu BAR(48) [ 3215.968907] workqueue ipv6_addrconf: flags=0x40008 [ 3215.974813] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3215.978684] pending: addrconf_verify_work [ 3215.982378] workqueue knbd0-recv: flags=0x1a [ 3215.989014] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3215.994776] in-flight: 650:recv_work [ 3215.997588] workqueue ptlrpc_pinger: flags=0x2 [ 3215.999324] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3216.001576] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3216.007130] pool 8: cpus=0-3 flags=0x4 nice=0 hung=702s workers=4 idle: 36 466 29757 [ 3216.016655] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3206s workers=2 idle: 652 [ 3224.059027] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 3224.062779] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3224.085289] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3224.091194] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3224.095774] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3224.098489] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3224.115023] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3224.122145] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3224.126863] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3224.132039] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3224.136135] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3224.139663] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3224.144324] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3224.148370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3224.152034] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3224.157331] Call Trace: [ 3224.159274] [ 3224.160684] ? show_regs.cold.9+0x22/0x2f [ 3224.163070] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3224.165721] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3224.168072] ? watchdog+0x40/0x40 [ 3224.170437] ? hrtimer_interrupt+0x128/0x2c0 [ 3224.172974] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3224.175367] ? apic_timer_interrupt+0xf/0x20 [ 3224.177651] [ 3224.179186] ? native_safe_halt+0x2e/0x50 [ 3224.181371] kvm_wait+0xcd/0xe0 [ 3224.183339] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3224.186006] do_raw_spin_lock+0x177/0x190 [ 3224.188201] _raw_spin_lock+0x21/0x30 [ 3224.190200] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3224.194112] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3224.196772] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3224.199739] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3224.203030] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3224.206128] ? woken_wake_function+0x30/0x30 [ 3224.208316] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3224.211933] kthread+0x1d1/0x200 [ 3224.213698] ? set_kthread_struct+0x70/0x70 [ 3224.215976] ret_from_fork+0x1f/0x30 [ 3232.060027] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 3232.064772] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3232.097503] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3232.114048] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3232.124349] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3232.137024] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3232.143980] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3232.156173] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3232.167188] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3232.175540] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3232.177899] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3232.184210] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3232.188864] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3232.192142] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3232.195621] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3232.198295] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3232.202108] Call Trace: [ 3232.203376] [ 3232.205663] ? show_regs.cold.9+0x22/0x2f [ 3232.208107] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3232.211101] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3232.216362] ? watchdog+0x40/0x40 [ 3232.218008] ? hrtimer_interrupt+0x128/0x2c0 [ 3232.220924] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3232.224282] ? apic_timer_interrupt+0xf/0x20 [ 3232.226535] [ 3232.227370] ? native_safe_halt+0x2e/0x50 [ 3232.231856] kvm_wait+0xcd/0xe0 [ 3232.235055] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3232.242029] do_raw_spin_lock+0x177/0x190 [ 3232.250319] _raw_spin_lock+0x21/0x30 [ 3232.255966] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3232.262146] process_one_work+0x2c8/0x700 [ 3232.266395] worker_thread+0x296/0x6e0 [ 3232.268116] ? rescuer_thread+0x570/0x570 [ 3232.269543] kthread+0x1d1/0x200 [ 3232.270620] ? set_kthread_struct+0x70/0x70 [ 3232.279710] ret_from_fork+0x1f/0x30 [ 3240.056827] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 3240.062019] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 3240.063587] Modules linked in: lustre(O) [ 3240.069585] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3240.069678] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3240.069686] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3240.069688] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3240.069702] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3240.069707] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3240.069717] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3240.069719] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3240.069730] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3240.069732] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3240.069735] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3240.069739] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3240.069742] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3240.069744] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3240.069752] Call Trace: [ 3240.069756] [ 3240.069762] ? show_regs.cold.9+0x22/0x2f [ 3240.069774] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3240.069780] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3240.069785] ? watchdog+0x40/0x40 [ 3240.069789] ? hrtimer_interrupt+0x128/0x2c0 [ 3240.069794] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3240.069800] ? apic_timer_interrupt+0xf/0x20 [ 3240.069805] [ 3240.069807] ? native_safe_halt+0x2e/0x50 [ 3240.069817] kvm_wait+0xcd/0xe0 [ 3240.069826] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3240.069837] do_raw_spin_lock+0x177/0x190 [ 3240.069841] _raw_spin_lock+0x21/0x30 [ 3240.069846] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3240.070271] ? do_raw_spin_unlock+0x75/0x190 [ 3240.070277] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3240.070763] ? do_wait_intr+0xf0/0xf0 [ 3240.070770] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3240.071012] mdc_reint+0x2d/0x110 [mdc] [ 3240.071057] mdc_unlink+0x658/0x870 [mdc] [ 3240.071333] lmv_unlink+0x34d/0xbd0 [lmv] [ 3240.071373] ll_unlink+0x24d/0x7f0 [lustre] [ 3240.071689] vfs_unlink+0x202/0x330 [ 3240.071701] do_unlinkat+0x2fa/0x460 [ 3240.071710] __x64_sys_unlink+0x37/0x50 [ 3240.071717] do_syscall_64+0xc1/0x440 [ 3240.071737] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3240.071743] RIP: 0033:0x7f83e5f5c10b [ 3240.071750] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3240.071759] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3240.071767] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3240.071770] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3240.071772] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3240.071774] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3240.071777] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3240.281014] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3240.310394] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3240.316293] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3240.320836] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3240.328894] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3240.339629] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3240.343053] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3240.345473] RDX: 0000000000000ced RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3240.351432] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3240.357122] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3240.361149] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3240.365589] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3240.373400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3240.376147] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3240.379648] Call Trace: [ 3240.380918] [ 3240.382006] ? show_regs.cold.9+0x22/0x2f [ 3240.384172] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3240.386403] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3240.388451] ? watchdog+0x40/0x40 [ 3240.390580] ? hrtimer_interrupt+0x128/0x2c0 [ 3240.396631] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3240.399269] ? apic_timer_interrupt+0xf/0x20 [ 3240.402575] [ 3240.403643] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3240.406793] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3240.409606] do_raw_spin_lock+0x177/0x190 [ 3240.411525] _raw_spin_lock+0x21/0x30 [ 3240.417525] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3240.421241] obd_mod_free+0x46/0x140 [ptlrpc] [ 3240.426338] mdc_commit_open+0x91/0xb0 [mdc] [ 3240.430191] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3240.435366] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3240.438595] after_reply+0xe2c/0x1780 [ptlrpc] [ 3240.441023] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3240.445131] ? __next_timer_interrupt+0x160/0x160 [ 3240.448464] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3240.450803] ? do_wait_intr+0xf0/0xf0 [ 3240.452897] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3240.455888] kthread+0x1d1/0x200 [ 3240.457477] ? set_kthread_struct+0x70/0x70 [ 3240.459479] ret_from_fork+0x1f/0x30 [ 3246.049027] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 732s! [ 3246.061077] Showing busy workqueues and worker pools: [ 3246.069630] workqueue events: flags=0x0 [ 3246.072064] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3246.077598] pending: osc_grant_work_handler [osc], push_to_pool [ 3246.084113] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3246.088994] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3246.098368] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3246.102992] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3246.112359] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3246.120521] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3246.126778] workqueue events_long: flags=0x0 [ 3246.131323] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3246.138894] pending: lu_objects_flush [obdclass] [ 3246.144491] workqueue events_unbound: flags=0x2 [ 3246.149312] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3246.152673] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3246.158614] workqueue events_power_efficient: flags=0x80 [ 3246.163277] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3246.168747] pending: neigh_periodic_work, do_cache_clean [ 3246.174498] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3246.180074] pending: neigh_periodic_work, check_lifetime [ 3246.182993] workqueue mm_percpu_wq: flags=0x8 [ 3246.185563] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3246.189400] pending: vmstat_update, lru_add_drain_per_cpu [ 3246.192358] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3246.197555] pending: vmstat_update [ 3246.202061] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3246.209128] pending: lru_add_drain_per_cpu BAR(48) [ 3246.212879] workqueue ipv6_addrconf: flags=0x40008 [ 3246.215831] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3246.224796] pending: addrconf_verify_work [ 3246.234476] workqueue knbd0-recv: flags=0x1a [ 3246.242606] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3246.249608] in-flight: 650:recv_work [ 3246.255190] workqueue ptlrpc_pinger: flags=0x2 [ 3246.260863] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3246.268337] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3246.272079] pool 8: cpus=0-3 flags=0x4 nice=0 hung=733s workers=4 idle: 36 466 29757 [ 3246.279293] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3236s workers=2 idle: 652 [ 3252.061674] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 3252.070863] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3252.124958] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3252.137457] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3252.141875] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3252.151645] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3252.161906] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3252.171621] RAX: 0000000000007c4b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3252.177569] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3252.192264] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3252.203396] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3252.210472] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3252.223103] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3252.235437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3252.243025] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3252.253885] Call Trace: [ 3252.254772] [ 3252.255458] ? show_regs.cold.9+0x22/0x2f [ 3252.259562] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3252.265071] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3252.268423] ? watchdog+0x40/0x40 [ 3252.271554] ? hrtimer_interrupt+0x128/0x2c0 [ 3252.275926] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3252.284027] ? apic_timer_interrupt+0xf/0x20 [ 3252.288890] [ 3252.290534] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3252.294434] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3252.298927] do_raw_spin_lock+0x177/0x190 [ 3252.302384] _raw_spin_lock+0x21/0x30 [ 3252.306450] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3252.312905] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3252.319002] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3252.327374] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3252.333804] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3252.338891] ? woken_wake_function+0x30/0x30 [ 3252.341257] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3252.350506] kthread+0x1d1/0x200 [ 3252.354082] ? set_kthread_struct+0x70/0x70 [ 3252.357468] ret_from_fork+0x1f/0x30 [ 3260.060015] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 3260.071927] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3260.175836] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3260.191766] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3260.202586] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3260.221197] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3260.238198] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3260.270153] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3260.282535] RAX: 00000000000046f8 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3260.303284] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3260.317866] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3260.333494] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3260.346789] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3260.350746] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3260.367389] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3260.382180] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3260.391575] Call Trace: [ 3260.392367] [ 3260.393016] ? show_regs.cold.9+0x22/0x2f [ 3260.400947] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3260.409951] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3260.413009] ? watchdog+0x40/0x40 [ 3260.420074] ? hrtimer_interrupt+0x128/0x2c0 [ 3260.421911] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3260.428955] ? apic_timer_interrupt+0xf/0x20 [ 3260.432097] [ 3260.437533] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3260.440732] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3260.446647] do_raw_spin_lock+0x177/0x190 [ 3260.451103] _raw_spin_lock+0x21/0x30 [ 3260.452781] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3260.456623] process_one_work+0x2c8/0x700 [ 3260.459377] worker_thread+0x296/0x6e0 [ 3260.460669] ? rescuer_thread+0x570/0x570 [ 3260.462119] kthread+0x1d1/0x200 [ 3260.471139] ? set_kthread_struct+0x70/0x70 [ 3260.480494] ret_from_fork+0x1f/0x30 [ 3268.065790] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 3268.086801] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3268.192784] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3268.216780] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3268.240213] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3268.247115] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3268.260474] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3268.280210] RAX: 0000000000007483 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3268.285486] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3268.295568] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3268.303191] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3268.307164] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3268.311216] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3268.315426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3268.318629] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3268.322288] Call Trace: [ 3268.323622] [ 3268.324590] ? show_regs.cold.9+0x22/0x2f [ 3268.326646] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3268.328798] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3268.330887] ? watchdog+0x40/0x40 [ 3268.332400] ? hrtimer_interrupt+0x128/0x2c0 [ 3268.334283] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3268.336596] ? apic_timer_interrupt+0xf/0x20 [ 3268.338652] [ 3268.339413] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3268.341335] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3268.343618] do_raw_spin_lock+0x177/0x190 [ 3268.345423] _raw_spin_lock+0x21/0x30 [ 3268.347328] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3268.349863] ? do_raw_spin_unlock+0x75/0x190 [ 3268.351922] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3268.354678] ? do_wait_intr+0xf0/0xf0 [ 3268.356346] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3268.358899] mdc_reint+0x2d/0x110 [mdc] [ 3268.360702] mdc_unlink+0x658/0x870 [mdc] [ 3268.362725] lmv_unlink+0x34d/0xbd0 [lmv] [ 3268.364641] ll_unlink+0x24d/0x7f0 [lustre] [ 3268.366813] vfs_unlink+0x202/0x330 [ 3268.368427] do_unlinkat+0x2fa/0x460 [ 3268.370069] __x64_sys_unlink+0x37/0x50 [ 3268.371859] do_syscall_64+0xc1/0x440 [ 3268.373524] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3268.375917] RIP: 0033:0x7f83e5f5c10b [ 3268.377612] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3268.386096] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3268.389625] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3268.392781] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3268.395876] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3268.399308] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3268.402672] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3272.058400] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 3272.064665] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3272.123822] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3272.138401] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3272.157477] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3272.167193] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3272.203478] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3272.213265] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3272.232271] RDX: 0000000000006d99 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3272.246947] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3272.257405] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3272.263220] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3272.278070] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3272.292302] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3272.300419] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3272.314996] Call Trace: [ 3272.317492] [ 3272.321236] ? show_regs.cold.9+0x22/0x2f [ 3272.328163] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3272.340076] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3272.342244] ? watchdog+0x40/0x40 [ 3272.345671] ? hrtimer_interrupt+0x128/0x2c0 [ 3272.354676] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3272.360823] ? apic_timer_interrupt+0xf/0x20 [ 3272.368178] [ 3272.369726] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3272.374094] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3272.377828] do_raw_spin_lock+0x177/0x190 [ 3272.380226] _raw_spin_lock+0x21/0x30 [ 3272.383293] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3272.386810] obd_mod_free+0x46/0x140 [ptlrpc] [ 3272.392552] mdc_commit_open+0x91/0xb0 [mdc] [ 3272.395652] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3272.403471] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3272.407379] after_reply+0xe2c/0x1780 [ptlrpc] [ 3272.412235] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3272.416149] ? __next_timer_interrupt+0x160/0x160 [ 3272.419816] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3272.439933] ? do_wait_intr+0xf0/0xf0 [ 3272.442849] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3272.450552] kthread+0x1d1/0x200 [ 3272.454463] ? set_kthread_struct+0x70/0x70 [ 3272.462049] ret_from_fork+0x1f/0x30 [ 3276.779551] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 763s! [ 3276.783582] Showing busy workqueues and worker pools: [ 3276.793362] workqueue events: flags=0x0 [ 3276.795269] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3276.804252] pending: osc_grant_work_handler [osc], push_to_pool [ 3276.811218] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3276.818506] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3276.823393] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3276.825124] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3276.844360] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3276.860084] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3276.867194] workqueue events_long: flags=0x0 [ 3276.873993] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3276.882529] pending: lu_objects_flush [obdclass] [ 3276.886664] workqueue events_unbound: flags=0x2 [ 3276.895099] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3276.909977] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3276.932925] workqueue events_power_efficient: flags=0x80 [ 3276.946745] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3276.962145] pending: neigh_periodic_work, do_cache_clean [ 3276.968948] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3276.971639] pending: neigh_periodic_work, check_lifetime [ 3276.983949] workqueue mm_percpu_wq: flags=0x8 [ 3276.985293] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3276.993999] pending: vmstat_update, lru_add_drain_per_cpu [ 3276.995603] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3277.017825] pending: vmstat_update [ 3277.019638] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3277.031944] pending: lru_add_drain_per_cpu BAR(48) [ 3277.038546] workqueue ipv6_addrconf: flags=0x40008 [ 3277.041150] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3277.046641] pending: addrconf_verify_work [ 3277.048896] workqueue knbd0-recv: flags=0x1a [ 3277.050883] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3277.055710] in-flight: 650:recv_work [ 3277.057964] workqueue ptlrpc_pinger: flags=0x2 [ 3277.061045] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3277.065675] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3277.070560] pool 8: cpus=0-3 flags=0x4 nice=0 hung=763s workers=4 idle: 36 466 29757 [ 3277.082324] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3267s workers=2 idle: 652 [ 3280.060473] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 3280.067980] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3280.126606] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3280.136433] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3280.147572] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3280.158926] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3280.175714] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3280.191524] RAX: 0000000000006f0d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3280.203313] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3280.214618] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3280.224067] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3280.232437] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3280.255072] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3280.266407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3280.276139] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3280.299092] Call Trace: [ 3280.300043] [ 3280.300721] ? show_regs.cold.9+0x22/0x2f [ 3280.311952] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3280.321711] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3280.325299] ? watchdog+0x40/0x40 [ 3280.333670] ? hrtimer_interrupt+0x128/0x2c0 [ 3280.349643] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3280.352532] ? apic_timer_interrupt+0xf/0x20 [ 3280.359486] [ 3280.364064] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3280.370644] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3280.385754] do_raw_spin_lock+0x177/0x190 [ 3280.401594] _raw_spin_lock+0x21/0x30 [ 3280.407536] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3280.422569] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3280.430982] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3280.439566] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3280.451284] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3280.462633] ? woken_wake_function+0x30/0x30 [ 3280.471886] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3280.486592] kthread+0x1d1/0x200 [ 3280.503747] ? set_kthread_struct+0x70/0x70 [ 3280.510959] ret_from_fork+0x1f/0x30 [ 3288.060063] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 3288.063868] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3288.085157] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3288.089957] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3288.094037] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3288.097486] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3288.099723] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3288.107987] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3288.110990] RAX: 0000000000000888 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3288.113553] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3288.116773] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3288.120006] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3288.122543] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3288.124674] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3288.128325] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3288.131211] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3288.134371] Call Trace: [ 3288.135528] [ 3288.136560] ? show_regs.cold.9+0x22/0x2f [ 3288.138507] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3288.140871] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3288.143365] ? watchdog+0x40/0x40 [ 3288.144808] ? hrtimer_interrupt+0x128/0x2c0 [ 3288.146883] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3288.149306] ? apic_timer_interrupt+0xf/0x20 [ 3288.151491] [ 3288.152415] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3288.154852] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3288.158099] do_raw_spin_lock+0x177/0x190 [ 3288.159920] _raw_spin_lock+0x21/0x30 [ 3288.161318] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3288.164130] process_one_work+0x2c8/0x700 [ 3288.165911] worker_thread+0x296/0x6e0 [ 3288.167504] ? rescuer_thread+0x570/0x570 [ 3288.169370] kthread+0x1d1/0x200 [ 3288.170432] ? set_kthread_struct+0x70/0x70 [ 3288.172447] ret_from_fork+0x1f/0x30 [ 3296.062059] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 3296.065821] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3296.088769] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3296.094664] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3296.099087] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3296.102290] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3296.111852] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3296.115166] RAX: 0000000000007683 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3296.118761] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3296.122096] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3296.126028] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3296.130147] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3296.134153] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3296.138430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3296.141163] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3296.144455] Call Trace: [ 3296.145550] [ 3296.146570] ? show_regs.cold.9+0x22/0x2f [ 3296.148726] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3296.151330] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3296.153664] ? watchdog+0x40/0x40 [ 3296.155835] ? hrtimer_interrupt+0x128/0x2c0 [ 3296.159570] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3296.162319] ? apic_timer_interrupt+0xf/0x20 [ 3296.166088] [ 3296.170331] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3296.177858] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3296.183971] do_raw_spin_lock+0x177/0x190 [ 3296.185837] _raw_spin_lock+0x21/0x30 [ 3296.194484] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3296.196539] ? do_raw_spin_unlock+0x75/0x190 [ 3296.201803] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3296.205160] ? do_wait_intr+0xf0/0xf0 [ 3296.207346] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3296.211213] mdc_reint+0x2d/0x110 [mdc] [ 3296.216347] mdc_unlink+0x658/0x870 [mdc] [ 3296.220490] lmv_unlink+0x34d/0xbd0 [lmv] [ 3296.225303] ll_unlink+0x24d/0x7f0 [lustre] [ 3296.228508] vfs_unlink+0x202/0x330 [ 3296.230193] do_unlinkat+0x2fa/0x460 [ 3296.232468] __x64_sys_unlink+0x37/0x50 [ 3296.234332] do_syscall_64+0xc1/0x440 [ 3296.236472] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3296.238944] RIP: 0033:0x7f83e5f5c10b [ 3296.240660] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3296.253774] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3296.260638] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3296.266168] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3296.277729] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3296.287298] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3296.291717] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3300.984074] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 3300.988699] rcu: 0-....: (838093 ticks this GP) idle=30a/1/0x4000000000000000 softirq=128519/128519 fqs=126395 [ 3300.994496] (detected by 1, t=1320025 jiffies, g=262441, q=57395) [ 3300.997645] Sending NMI from CPU 1 to CPUs 0: [ 3301.009202] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 7.463 msecs [ 3301.009605] NMI backtrace for cpu 0 [ 3301.009608] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3301.009610] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3301.009611] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3301.009613] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3301.009615] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 [ 3301.009617] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3301.009619] RDX: 0000000000006e1d RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3301.009620] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3301.009621] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3301.009622] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3301.009624] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3301.009625] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3301.009626] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3301.009627] Call Trace: [ 3301.009628] [ 3301.009629] ? show_regs.cold.9+0x22/0x2f [ 3301.009630] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3301.009631] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3301.009632] ? nmi_handle+0xbd/0x210 [ 3301.009633] ? default_do_nmi+0x4b/0x1b0 [ 3301.009634] ? do_nmi+0x280/0x380 [ 3301.009635] ? end_repeat_nmi+0x16/0x1f [ 3301.009636] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3301.009637] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3301.009638] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3301.009639] [ 3301.009640] do_raw_spin_lock+0x177/0x190 [ 3301.009641] _raw_spin_lock+0x21/0x30 [ 3301.009642] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3301.009643] obd_mod_free+0x46/0x140 [ptlrpc] [ 3301.009645] mdc_commit_open+0x91/0xb0 [mdc] [ 3301.009646] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3301.009647] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3301.009648] after_reply+0xe2c/0x1780 [ptlrpc] [ 3301.009649] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3301.009650] ? __next_timer_interrupt+0x160/0x160 [ 3301.009651] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3301.009652] ? do_wait_intr+0xf0/0xf0 [ 3301.009653] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3301.009654] kthread+0x1d1/0x200 [ 3301.009655] ? set_kthread_struct+0x70/0x70 [ 3301.009656] ret_from_fork+0x1f/0x30 [ 3301.009661] rcu: rcu_sched kthread starved for 789212 jiffies! g262441 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 3301.161067] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3301.166651] rcu: RCU grace-period kthread stack dump: [ 3301.169200] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3301.177631] Call Trace: [ 3301.178776] __schedule+0x351/0xcb0 [ 3301.181228] schedule+0xc0/0x180 [ 3301.182618] schedule_timeout+0xb4/0x190 [ 3301.184555] ? do_raw_spin_unlock+0x75/0x190 [ 3301.186703] ? __next_timer_interrupt+0x160/0x160 [ 3301.189019] ? prepare_to_swait_event+0xf3/0x140 [ 3301.191106] rcu_gp_kthread+0x8dd/0x15b0 [ 3301.192766] ? force_qs_rnp+0x2c0/0x2c0 [ 3301.194371] kthread+0x1d1/0x200 [ 3301.195539] ? set_kthread_struct+0x70/0x70 [ 3301.197510] ret_from_fork+0x1f/0x30 [ 3301.198517] rcu: Stack dump where RCU GP kthread last ran: [ 3301.200772] Sending NMI from CPU 1 to CPUs 3: [ 3301.204356] NMI backtrace for cpu 3 [ 3301.204368] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3301.204370] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3301.204371] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3301.204373] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3301.204375] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 [ 3301.204377] RAX: 0000000000006c14 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3301.204378] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3301.204380] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3301.204381] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3301.204382] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3301.204383] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3301.204384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3301.204385] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3301.204386] Call Trace: [ 3301.204387] [ 3301.204388] ? show_regs.cold.9+0x22/0x2f [ 3301.204389] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3301.204390] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3301.204390] ? nmi_handle+0xbd/0x210 [ 3301.204391] ? default_do_nmi+0x4b/0x1b0 [ 3301.204392] ? do_nmi+0x280/0x380 [ 3301.204393] ? end_repeat_nmi+0x16/0x1f [ 3301.204394] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3301.204395] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3301.204396] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3301.204397] [ 3301.204398] do_raw_spin_lock+0x177/0x190 [ 3301.204399] _raw_spin_lock+0x21/0x30 [ 3301.204399] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3301.204401] ? do_raw_spin_unlock+0x75/0x190 [ 3301.204401] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3301.204402] ? do_wait_intr+0xf0/0xf0 [ 3301.204403] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3301.204404] mdc_reint+0x2d/0x110 [mdc] [ 3301.204405] mdc_unlink+0x658/0x870 [mdc] [ 3301.204405] lmv_unlink+0x34d/0xbd0 [lmv] [ 3301.204406] ll_unlink+0x24d/0x7f0 [lustre] [ 3301.204407] vfs_unlink+0x202/0x330 [ 3301.204408] do_unlinkat+0x2fa/0x460 [ 3301.204409] __x64_sys_unlink+0x37/0x50 [ 3301.204410] do_syscall_64+0xc1/0x440 [ 3301.204411] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3301.204412] RIP: 0033:0x7f83e5f5c10b [ 3301.204413] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3301.204415] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3301.204417] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3301.204418] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3301.204420] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3301.204421] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3301.204422] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3307.489586] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 794s! [ 3307.493748] Showing busy workqueues and worker pools: [ 3307.496100] workqueue events: flags=0x0 [ 3307.497889] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3307.500655] pending: osc_grant_work_handler [osc], push_to_pool [ 3307.503795] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3307.506546] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3307.509385] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3307.512280] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3307.515592] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3307.518813] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3307.523018] workqueue events_long: flags=0x0 [ 3307.524167] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3307.525805] pending: lu_objects_flush [obdclass] [ 3307.528305] workqueue events_unbound: flags=0x2 [ 3307.529827] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3307.532367] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3307.537626] workqueue events_power_efficient: flags=0x80 [ 3307.539630] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3307.541992] pending: neigh_periodic_work, do_cache_clean [ 3307.544064] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3307.547466] pending: neigh_periodic_work, check_lifetime [ 3307.552386] workqueue mm_percpu_wq: flags=0x8 [ 3307.553692] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3307.557404] pending: vmstat_update, lru_add_drain_per_cpu [ 3307.568985] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3307.581282] pending: vmstat_update [ 3307.583525] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3307.589593] pending: lru_add_drain_per_cpu BAR(48) [ 3307.591622] workqueue ipv6_addrconf: flags=0x40008 [ 3307.599298] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3307.601764] pending: addrconf_verify_work [ 3307.605523] workqueue knbd0-recv: flags=0x1a [ 3307.608309] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3307.614257] in-flight: 650:recv_work [ 3307.618663] workqueue ptlrpc_pinger: flags=0x2 [ 3307.621893] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3307.625990] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3307.629148] pool 8: cpus=0-3 flags=0x4 nice=0 hung=794s workers=4 idle: 36 466 29757 [ 3307.632945] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3297s workers=2 idle: 652 [ 3316.060068] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 3316.064214] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3316.094117] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3316.106483] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3316.117286] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3316.122946] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3316.128206] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3316.145478] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3316.150798] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3316.154834] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3316.158398] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3316.162394] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3316.167439] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3316.170923] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3316.182250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3316.186407] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3316.192570] Call Trace: [ 3316.194651] [ 3316.195752] ? show_regs.cold.9+0x22/0x2f [ 3316.200826] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3316.206539] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3316.208771] ? watchdog+0x40/0x40 [ 3316.210734] ? hrtimer_interrupt+0x128/0x2c0 [ 3316.212223] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3316.221417] ? apic_timer_interrupt+0xf/0x20 [ 3316.222768] [ 3316.236034] ? native_safe_halt+0x2e/0x50 [ 3316.237195] kvm_wait+0xcd/0xe0 [ 3316.238150] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3316.246385] do_raw_spin_lock+0x177/0x190 [ 3316.250051] _raw_spin_lock+0x21/0x30 [ 3316.252644] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3316.257608] process_one_work+0x2c8/0x700 [ 3316.263377] worker_thread+0x296/0x6e0 [ 3316.265276] ? rescuer_thread+0x570/0x570 [ 3316.266673] kthread+0x1d1/0x200 [ 3316.272230] ? set_kthread_struct+0x70/0x70 [ 3316.276397] ret_from_fork+0x1f/0x30 [ 3328.059286] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 3328.062012] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 3328.065755] Modules linked in: [ 3328.069821] Modules linked in: lustre(O) osp(O) [ 3328.071479] lustre(O) osp(O) ofd(O) lod(O) mdt(O) [ 3328.074157] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3328.074229] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3328.074236] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3328.074239] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3328.074255] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3328.074259] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3328.074267] RAX: 0000000000002047 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3328.074270] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3328.076499] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) [ 3328.106286] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3328.106296] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3328.106299] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3328.106302] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3328.106305] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3328.106308] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3328.106316] Call Trace: [ 3328.106322] [ 3328.106328] ? show_regs.cold.9+0x22/0x2f [ 3328.106341] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3328.106347] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3328.106351] ? watchdog+0x40/0x40 [ 3328.106355] ? hrtimer_interrupt+0x128/0x2c0 [ 3328.106358] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3328.106366] ? apic_timer_interrupt+0xf/0x20 [ 3328.106370] [ 3328.106373] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3328.106380] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3328.106384] do_raw_spin_lock+0x177/0x190 [ 3328.106389] _raw_spin_lock+0x21/0x30 [ 3328.106396] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3328.106684] ? do_raw_spin_unlock+0x75/0x190 [ 3328.106691] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3328.106884] ? do_wait_intr+0xf0/0xf0 [ 3328.106893] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3328.107301] mdc_reint+0x2d/0x110 [mdc] [ 3328.107340] mdc_unlink+0x658/0x870 [mdc] [ 3328.107366] lmv_unlink+0x34d/0xbd0 [lmv] [ 3328.107406] ll_unlink+0x24d/0x7f0 [lustre] [ 3328.107524] vfs_unlink+0x202/0x330 [ 3328.107532] do_unlinkat+0x2fa/0x460 [ 3328.107537] __x64_sys_unlink+0x37/0x50 [ 3328.107540] do_syscall_64+0xc1/0x440 [ 3328.107547] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3328.107567] RIP: 0033:0x7f83e5f5c10b [ 3328.107577] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3328.107581] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3328.107586] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3328.107588] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3328.107591] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3328.107593] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3328.107595] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3328.238457] osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3328.252414] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3328.257841] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3328.261323] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3328.266290] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3328.274728] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3328.277747] RAX: 0000000000006479 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3328.281505] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3328.285291] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3328.288633] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3328.294165] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3328.297981] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3328.303346] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3328.306340] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3328.309940] Call Trace: [ 3328.311332] [ 3328.312496] ? show_regs.cold.9+0x22/0x2f [ 3328.314020] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3328.315830] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3328.317682] ? watchdog+0x40/0x40 [ 3328.318941] ? hrtimer_interrupt+0x128/0x2c0 [ 3328.320849] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3328.322836] ? apic_timer_interrupt+0xf/0x20 [ 3328.324747] [ 3328.325453] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3328.327778] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3328.330337] do_raw_spin_lock+0x177/0x190 [ 3328.332339] _raw_spin_lock+0x21/0x30 [ 3328.333527] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3328.337511] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3328.339752] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3328.342567] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3328.345232] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3328.347415] ? woken_wake_function+0x30/0x30 [ 3328.349134] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3328.351800] kthread+0x1d1/0x200 [ 3328.353464] ? set_kthread_struct+0x70/0x70 [ 3328.355110] ret_from_fork+0x1f/0x30 [ 3332.056067] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 3332.062891] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3332.103059] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3332.110244] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3332.118139] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3332.121108] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3332.133084] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3332.141279] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3332.146032] RDX: 0000000000006b33 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3332.151895] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3332.158571] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3332.163515] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3332.168612] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3332.174474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3332.179405] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3332.184349] Call Trace: [ 3332.186702] [ 3332.188333] ? show_regs.cold.9+0x22/0x2f [ 3332.191874] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3332.195913] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3332.198641] ? watchdog+0x40/0x40 [ 3332.200669] ? hrtimer_interrupt+0x128/0x2c0 [ 3332.203866] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3332.206661] ? apic_timer_interrupt+0xf/0x20 [ 3332.209266] [ 3332.210606] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3332.215592] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3332.223454] do_raw_spin_lock+0x177/0x190 [ 3332.226455] _raw_spin_lock+0x21/0x30 [ 3332.229672] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3332.235470] obd_mod_free+0x46/0x140 [ptlrpc] [ 3332.239789] mdc_commit_open+0x91/0xb0 [mdc] [ 3332.244795] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3332.248334] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3332.253456] after_reply+0xe2c/0x1780 [ptlrpc] [ 3332.257303] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3332.261082] ? __next_timer_interrupt+0x160/0x160 [ 3332.263852] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3332.268310] ? do_wait_intr+0xf0/0xf0 [ 3332.270468] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3332.274314] kthread+0x1d1/0x200 [ 3332.276802] ? set_kthread_struct+0x70/0x70 [ 3332.279184] ret_from_fork+0x1f/0x30 [ 3337.697106] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 824s! [ 3337.702648] Showing busy workqueues and worker pools: [ 3337.705851] workqueue events: flags=0x0 [ 3337.707312] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3337.711710] pending: osc_grant_work_handler [osc], push_to_pool [ 3337.725685] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3337.731442] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3337.735477] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3337.737943] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3337.742011] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3337.746855] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3337.760605] workqueue events_long: flags=0x0 [ 3337.767927] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3337.771714] pending: lu_objects_flush [obdclass] [ 3337.777175] workqueue events_unbound: flags=0x2 [ 3337.779459] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3337.786737] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3337.790726] workqueue events_power_efficient: flags=0x80 [ 3337.800042] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3337.804923] pending: neigh_periodic_work, do_cache_clean [ 3337.813208] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3337.817941] pending: neigh_periodic_work, check_lifetime [ 3337.822596] workqueue mm_percpu_wq: flags=0x8 [ 3337.827846] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3337.843705] pending: vmstat_update, lru_add_drain_per_cpu [ 3337.849008] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3337.857971] pending: vmstat_update [ 3337.859338] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3337.871137] pending: lru_add_drain_per_cpu BAR(48) [ 3337.878723] workqueue ipv6_addrconf: flags=0x40008 [ 3337.880486] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3337.883548] pending: addrconf_verify_work [ 3337.890130] workqueue knbd0-recv: flags=0x1a [ 3337.892945] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3337.897153] in-flight: 650:recv_work [ 3337.900106] workqueue ptlrpc_pinger: flags=0x2 [ 3337.907900] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3337.911679] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3337.915965] pool 8: cpus=0-3 flags=0x4 nice=0 hung=824s workers=4 idle: 36 466 29757 [ 3337.921716] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3327s workers=2 idle: 652 [ 3344.061263] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3344.071145] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3344.113175] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3344.121972] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3344.128995] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3344.137419] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3344.144184] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3344.159395] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3344.174286] RAX: 0000000000007f86 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3344.179093] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3344.188407] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3344.197432] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3344.204460] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3344.215848] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3344.222804] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3344.229988] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3344.237146] Call Trace: [ 3344.238464] [ 3344.239547] ? show_regs.cold.9+0x22/0x2f [ 3344.245622] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3344.248278] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3344.250879] ? watchdog+0x40/0x40 [ 3344.254411] ? hrtimer_interrupt+0x128/0x2c0 [ 3344.256907] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3344.259524] ? apic_timer_interrupt+0xf/0x20 [ 3344.264336] [ 3344.267728] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3344.273528] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3344.276403] do_raw_spin_lock+0x177/0x190 [ 3344.280080] _raw_spin_lock+0x21/0x30 [ 3344.282105] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3344.285039] process_one_work+0x2c8/0x700 [ 3344.287328] worker_thread+0x296/0x6e0 [ 3344.289456] ? rescuer_thread+0x570/0x570 [ 3344.291264] kthread+0x1d1/0x200 [ 3344.292930] ? set_kthread_struct+0x70/0x70 [ 3344.294866] ret_from_fork+0x1f/0x30 [ 3356.059041] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 3356.063021] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3356.063342] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 3356.081382] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3356.081404] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3356.081407] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3356.081425] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3356.081429] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3356.081436] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3356.081438] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3356.081440] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3356.081442] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3356.081445] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3356.081448] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3356.081451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3356.081453] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3356.081459] Call Trace: [ 3356.081465] [ 3356.081470] ? show_regs.cold.9+0x22/0x2f [ 3356.081478] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3356.081483] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3356.081487] ? watchdog+0x40/0x40 [ 3356.081491] ? hrtimer_interrupt+0x128/0x2c0 [ 3356.081495] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3356.081506] ? apic_timer_interrupt+0xf/0x20 [ 3356.081511] [ 3356.081513] ? native_safe_halt+0x2e/0x50 [ 3356.081516] kvm_wait+0xcd/0xe0 [ 3356.081523] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3356.081532] do_raw_spin_lock+0x177/0x190 [ 3356.081535] _raw_spin_lock+0x21/0x30 [ 3356.081539] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3356.081835] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3356.082029] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3356.082372] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3356.082644] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3356.082869] ? woken_wake_function+0x30/0x30 [ 3356.082878] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3356.083451] kthread+0x1d1/0x200 [ 3356.083457] ? set_kthread_struct+0x70/0x70 [ 3356.083461] ret_from_fork+0x1f/0x30 [ 3356.275414] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3356.295579] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3356.299890] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3356.302959] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3356.304833] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3356.312823] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3356.316355] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3356.321901] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3356.328980] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3356.331238] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3356.340738] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3356.350318] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3356.356966] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3356.359118] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3356.368023] Call Trace: [ 3356.368964] [ 3356.369737] ? show_regs.cold.9+0x22/0x2f [ 3356.378691] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3356.380266] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3356.382424] ? watchdog+0x40/0x40 [ 3356.383759] ? hrtimer_interrupt+0x128/0x2c0 [ 3356.386969] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3356.400424] ? apic_timer_interrupt+0xf/0x20 [ 3356.405158] [ 3356.410131] ? native_safe_halt+0x2e/0x50 [ 3356.420007] kvm_wait+0xcd/0xe0 [ 3356.424185] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3356.430852] do_raw_spin_lock+0x177/0x190 [ 3356.432350] _raw_spin_lock+0x21/0x30 [ 3356.440911] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3356.455893] ? do_raw_spin_unlock+0x75/0x190 [ 3356.459941] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3356.466434] ? do_wait_intr+0xf0/0xf0 [ 3356.468062] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3356.470463] mdc_reint+0x2d/0x110 [mdc] [ 3356.476506] mdc_unlink+0x658/0x870 [mdc] [ 3356.486750] lmv_unlink+0x34d/0xbd0 [lmv] [ 3356.494878] ll_unlink+0x24d/0x7f0 [lustre] [ 3356.502032] vfs_unlink+0x202/0x330 [ 3356.507007] do_unlinkat+0x2fa/0x460 [ 3356.508962] __x64_sys_unlink+0x37/0x50 [ 3356.510340] do_syscall_64+0xc1/0x440 [ 3356.511677] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3356.523257] RIP: 0033:0x7f83e5f5c10b [ 3356.526281] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3356.554991] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3356.558778] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3356.564050] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3356.569051] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3356.572340] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3356.577182] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3364.056029] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 3364.059006] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3364.076584] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3364.080977] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3364.084371] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3364.086030] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3364.092925] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3364.095926] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3364.098665] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3364.101396] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3364.104300] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3364.107045] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3364.109832] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3364.113078] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3364.115338] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3364.118094] Call Trace: [ 3364.119084] [ 3364.119919] ? show_regs.cold.9+0x22/0x2f [ 3364.121478] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3364.126865] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3364.128727] ? watchdog+0x40/0x40 [ 3364.130046] ? hrtimer_interrupt+0x128/0x2c0 [ 3364.131753] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3364.133603] ? apic_timer_interrupt+0xf/0x20 [ 3364.135179] [ 3364.136028] ? native_safe_halt+0x2e/0x50 [ 3364.137510] kvm_wait+0xcd/0xe0 [ 3364.138832] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3364.140925] do_raw_spin_lock+0x177/0x190 [ 3364.142641] _raw_spin_lock+0x21/0x30 [ 3364.144018] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3364.146252] obd_mod_free+0x46/0x140 [ptlrpc] [ 3364.148379] mdc_commit_open+0x91/0xb0 [mdc] [ 3364.150193] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3364.152418] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3364.154583] after_reply+0xe2c/0x1780 [ptlrpc] [ 3364.156579] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3364.158797] ? __next_timer_interrupt+0x160/0x160 [ 3364.160680] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3364.162631] ? do_wait_intr+0xf0/0xf0 [ 3364.164152] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3364.166360] kthread+0x1d1/0x200 [ 3364.167636] ? set_kthread_struct+0x70/0x70 [ 3364.169302] ret_from_fork+0x1f/0x30 [ 3368.418659] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 855s! [ 3368.422774] Showing busy workqueues and worker pools: [ 3368.424841] workqueue events: flags=0x0 [ 3368.426660] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3368.430681] pending: osc_grant_work_handler [osc], push_to_pool [ 3368.434848] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3368.449429] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3368.451688] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3368.456976] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3368.462742] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3368.476069] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3368.486287] workqueue events_long: flags=0x0 [ 3368.487587] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3368.491777] pending: lu_objects_flush [obdclass] [ 3368.494683] workqueue events_unbound: flags=0x2 [ 3368.496975] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3368.499963] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3368.501879] workqueue events_power_efficient: flags=0x80 [ 3368.512027] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3368.519664] pending: neigh_periodic_work, do_cache_clean [ 3368.524034] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3368.527874] pending: neigh_periodic_work, check_lifetime [ 3368.532456] workqueue mm_percpu_wq: flags=0x8 [ 3368.535122] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3368.539860] pending: vmstat_update, lru_add_drain_per_cpu [ 3368.543601] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3368.547010] pending: vmstat_update [ 3368.548957] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3368.556770] pending: lru_add_drain_per_cpu BAR(48) [ 3368.560053] workqueue ipv6_addrconf: flags=0x40008 [ 3368.564925] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3368.572936] pending: addrconf_verify_work [ 3368.577793] workqueue knbd0-recv: flags=0x1a [ 3368.588429] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3368.591723] in-flight: 650:recv_work [ 3368.598680] workqueue ptlrpc_pinger: flags=0x2 [ 3368.600963] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3368.604089] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3368.607139] pool 8: cpus=0-3 flags=0x4 nice=0 hung=855s workers=4 idle: 36 466 29757 [ 3368.614292] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3358s workers=2 idle: 652 [ 3372.060031] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3372.062634] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3372.111826] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3372.118013] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3372.122854] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3372.126282] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3372.128568] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3372.139360] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3372.151324] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3372.158539] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3372.166287] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3372.174669] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3372.180881] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3372.189389] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3372.195708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3372.198656] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3372.202298] Call Trace: [ 3372.203430] [ 3372.206958] ? show_regs.cold.9+0x22/0x2f [ 3372.214157] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3372.231444] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3372.233056] ? watchdog+0x40/0x40 [ 3372.234226] ? hrtimer_interrupt+0x128/0x2c0 [ 3372.251263] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3372.264076] ? apic_timer_interrupt+0xf/0x20 [ 3372.267236] [ 3372.269382] ? native_safe_halt+0x2e/0x50 [ 3372.271079] kvm_wait+0xcd/0xe0 [ 3372.272393] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3372.274473] do_raw_spin_lock+0x177/0x190 [ 3372.276072] _raw_spin_lock+0x21/0x30 [ 3372.278074] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3372.281023] process_one_work+0x2c8/0x700 [ 3372.282458] worker_thread+0x296/0x6e0 [ 3372.283919] ? rescuer_thread+0x570/0x570 [ 3372.285433] kthread+0x1d1/0x200 [ 3372.286649] ? set_kthread_struct+0x70/0x70 [ 3372.288171] ret_from_fork+0x1f/0x30 [ 3384.062013] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 3384.064264] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 3384.066052] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3384.066143] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3384.066150] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3384.066153] RIP: 0010:dev_gro_receive+0x65a/0x8b0 [ 3384.066167] Code: 00 48 83 05 af 64 ea 02 01 31 d2 e9 38 01 00 00 48 83 05 a8 21 ea 02 01 bd 03 00 00 00 e9 f6 f9 ff ff 48 83 05 d6 21 ea 02 01 e4 f9 ff ff 45 31 d2 48 83 05 06 c9 e9 02 01 4d 8d 48 f8 4d 8b [ 3384.066171] RSP: 0018:ffffbc1d00d8cd28 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3384.066176] RAX: 0000000000000000 RBX: ffff9a61b999e900 RCX: 0000000000000000 [ 3384.066178] RDX: ffff9a61b999cc4e RSI: 0000000000000028 RDI: 0000000000000028 [ 3384.066180] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000076 [ 3384.066183] R10: 0000000000000000 R11: ffffbc1d00d8ce18 R12: ffff9a61a19a08c8 [ 3384.066185] R13: 0000000000000000 R14: ffff9a61a19a0808 R15: 0000000000000000 [ 3384.066188] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3384.066191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3384.066193] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3384.066201] Call Trace: [ 3384.066207] [ 3384.066213] ? show_regs.cold.9+0x22/0x2f [ 3384.066222] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3384.066230] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3384.066234] ? watchdog+0x40/0x40 [ 3384.066238] ? hrtimer_interrupt+0x128/0x2c0 [ 3384.066241] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3384.066249] ? apic_timer_interrupt+0xf/0x20 [ 3384.066254] ? apic_timer_interrupt+0xa/0x20 [ 3384.066259] ? dev_gro_receive+0x65a/0x8b0 [ 3384.066264] ? dev_gro_receive+0x314/0x8b0 [ 3384.066269] napi_gro_receive+0xa0/0x2a0 [ 3384.066274] receive_buf+0x31f/0xad0 [ 3384.066280] ? detach_buf_split+0x94/0x1f0 [ 3384.066287] virtnet_poll+0x27c/0x628 [ 3384.066293] __napi_poll+0x37/0x260 [ 3384.066298] net_rx_action+0x35c/0x470 [ 3384.066303] ? vp_interrupt+0x3a/0x70 [ 3384.066307] __do_softirq+0x134/0x4e8 [ 3384.066313] irq_exit_rcu+0x1c6/0x1e0 [ 3384.066319] irq_exit+0x12/0x30 [ 3384.066323] do_IRQ+0x62/0x120 [ 3384.066329] common_interrupt+0xf/0xf [ 3384.066334] [ 3384.066335] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3384.066342] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3384.066346] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffffd9 [ 3384.066349] RAX: 0000000000000488 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3384.066351] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3384.066353] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3384.066355] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3384.066357] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3384.066361] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3384.066367] do_raw_spin_lock+0x177/0x190 [ 3384.066372] _raw_spin_lock+0x21/0x30 [ 3384.066377] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3384.071871] Modules linked in: lustre(O) [ 3384.102416] ? do_raw_spin_unlock+0x75/0x190 [ 3384.113487] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3384.113624] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3384.113631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3384.113633] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3384.113647] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3384.113651] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3384.113656] RAX: 0000000000001d84 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3384.113659] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3384.113661] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3384.113663] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3384.113665] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3384.113669] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3384.113672] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3384.113674] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3384.113682] Call Trace: [ 3384.113688] [ 3384.113694] ? show_regs.cold.9+0x22/0x2f [ 3384.113704] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3384.113715] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3384.113719] ? watchdog+0x40/0x40 [ 3384.113722] ? hrtimer_interrupt+0x128/0x2c0 [ 3384.113726] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3384.113732] ? apic_timer_interrupt+0xf/0x20 [ 3384.113736] [ 3384.113794] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3384.113800] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3384.113806] do_raw_spin_lock+0x177/0x190 [ 3384.113810] _raw_spin_lock+0x21/0x30 [ 3384.113815] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3384.114159] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3384.114473] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3384.115030] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3384.115264] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3384.115688] ? woken_wake_function+0x30/0x30 [ 3384.115699] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3384.116049] kthread+0x1d1/0x200 [ 3384.116055] ? set_kthread_struct+0x70/0x70 [ 3384.116060] ret_from_fork+0x1f/0x30 [ 3384.561875] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3384.563277] ? do_wait_intr+0xf0/0xf0 [ 3384.564105] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3384.584801] mdc_reint+0x2d/0x110 [mdc] [ 3384.586979] mdc_unlink+0x658/0x870 [mdc] [ 3384.591691] lmv_unlink+0x34d/0xbd0 [lmv] [ 3384.597186] ll_unlink+0x24d/0x7f0 [lustre] [ 3384.603361] vfs_unlink+0x202/0x330 [ 3384.605969] do_unlinkat+0x2fa/0x460 [ 3384.607854] __x64_sys_unlink+0x37/0x50 [ 3384.609438] do_syscall_64+0xc1/0x440 [ 3384.611111] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3384.612921] RIP: 0033:0x7f83e5f5c10b [ 3384.614314] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3384.621923] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3384.625903] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3384.629613] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3384.632847] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3384.638140] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3384.649308] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3396.056022] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 3396.061665] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3396.170337] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3396.181169] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3396.189311] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3396.195405] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3396.235503] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3396.256579] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3396.273893] RDX: 000000000000319c RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3396.292332] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3396.301399] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3396.308559] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3396.330614] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3396.350556] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3396.355835] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3396.375287] Call Trace: [ 3396.381451] [ 3396.382715] ? show_regs.cold.9+0x22/0x2f [ 3396.390577] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3396.396918] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3396.404403] ? watchdog+0x40/0x40 [ 3396.407127] ? hrtimer_interrupt+0x128/0x2c0 [ 3396.410991] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3396.426232] ? apic_timer_interrupt+0xf/0x20 [ 3396.427712] [ 3396.428429] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3396.437601] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3396.442629] do_raw_spin_lock+0x177/0x190 [ 3396.446127] _raw_spin_lock+0x21/0x30 [ 3396.451284] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3396.460647] obd_mod_free+0x46/0x140 [ptlrpc] [ 3396.470405] mdc_commit_open+0x91/0xb0 [mdc] [ 3396.473515] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3396.477606] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3396.486655] after_reply+0xe2c/0x1780 [ptlrpc] [ 3396.495401] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3396.502837] ? __next_timer_interrupt+0x160/0x160 [ 3396.509275] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3396.516838] ? do_wait_intr+0xf0/0xf0 [ 3396.519601] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3396.526793] kthread+0x1d1/0x200 [ 3396.528026] ? set_kthread_struct+0x70/0x70 [ 3396.533027] ret_from_fork+0x1f/0x30 [ 3398.632151] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 885s! [ 3398.637638] Showing busy workqueues and worker pools: [ 3398.642120] workqueue events: flags=0x0 [ 3398.645130] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3398.650760] pending: osc_grant_work_handler [osc], push_to_pool [ 3398.656109] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3398.661164] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3398.666772] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3398.671609] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3398.679358] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3398.685192] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3398.691773] workqueue events_long: flags=0x0 [ 3398.695340] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3398.705753] pending: lu_objects_flush [obdclass] [ 3398.711569] workqueue events_unbound: flags=0x2 [ 3398.715076] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3398.720533] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3398.724569] workqueue events_power_efficient: flags=0x80 [ 3398.728502] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3398.733243] pending: neigh_periodic_work, do_cache_clean [ 3398.738032] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3398.742548] pending: neigh_periodic_work, check_lifetime [ 3398.747830] workqueue mm_percpu_wq: flags=0x8 [ 3398.751643] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3398.757673] pending: vmstat_update, lru_add_drain_per_cpu [ 3398.761516] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3398.766090] pending: vmstat_update [ 3398.768746] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3398.773544] pending: lru_add_drain_per_cpu BAR(48) [ 3398.775463] workqueue ipv6_addrconf: flags=0x40008 [ 3398.779339] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3398.789854] pending: addrconf_verify_work [ 3398.795198] workqueue knbd0-recv: flags=0x1a [ 3398.798453] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3398.803434] in-flight: 650:recv_work [ 3398.806513] workqueue ptlrpc_pinger: flags=0x2 [ 3398.808152] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3398.811784] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3398.816697] pool 8: cpus=0-3 flags=0x4 nice=0 hung=885s workers=4 idle: 36 466 29757 [ 3398.821344] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3388s workers=2 idle: 652 [ 3400.060025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3400.063407] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3400.128663] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3400.153716] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3400.173520] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3400.187610] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3400.201454] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3400.255451] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3400.261831] RAX: 0000000000001302 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3400.268477] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3400.273447] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3400.286595] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3400.292457] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3400.302689] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3400.319256] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3400.330304] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3400.349851] Call Trace: [ 3400.352038] [ 3400.353124] ? show_regs.cold.9+0x22/0x2f [ 3400.364448] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3400.371120] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3400.380199] ? watchdog+0x40/0x40 [ 3400.386076] ? hrtimer_interrupt+0x128/0x2c0 [ 3400.393343] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3400.400067] ? apic_timer_interrupt+0xf/0x20 [ 3400.410056] [ 3400.412491] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3400.426264] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3400.430782] do_raw_spin_lock+0x177/0x190 [ 3400.444522] _raw_spin_lock+0x21/0x30 [ 3400.447401] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3400.463134] process_one_work+0x2c8/0x700 [ 3400.473606] worker_thread+0x296/0x6e0 [ 3400.481518] ? rescuer_thread+0x570/0x570 [ 3400.492211] kthread+0x1d1/0x200 [ 3400.493365] ? set_kthread_struct+0x70/0x70 [ 3400.513596] ret_from_fork+0x1f/0x30 [ 3412.062419] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 3412.062702] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 3412.072402] Modules linked in: lustre(O) osp(O) ofd(O) [ 3412.080991] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3412.081078] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3412.081085] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3412.081088] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3412.081101] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3412.081105] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3412.081110] RAX: 0000000000007683 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3412.081112] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3412.081114] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3412.081116] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3412.081119] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3412.081122] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3412.081125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3412.081127] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3412.081136] Call Trace: [ 3412.081141] [ 3412.081147] ? show_regs.cold.9+0x22/0x2f [ 3412.081158] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3412.081165] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3412.081169] ? watchdog+0x40/0x40 [ 3412.081174] ? hrtimer_interrupt+0x128/0x2c0 [ 3412.081190] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3412.081200] ? apic_timer_interrupt+0xf/0x20 [ 3412.081204] [ 3412.081207] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3412.081212] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3412.081217] do_raw_spin_lock+0x177/0x190 [ 3412.081221] _raw_spin_lock+0x21/0x30 [ 3412.081227] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3412.091806] lod(O) [ 3412.116371] ? do_raw_spin_unlock+0x75/0x190 [ 3412.116396] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3412.120881] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3412.121030] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3412.121038] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3412.121041] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3412.121134] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3412.121139] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3412.121143] RAX: 0000000000007990 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3412.121145] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3412.121147] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3412.121149] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3412.121151] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3412.121155] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3412.121157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3412.121160] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3412.121168] Call Trace: [ 3412.121175] [ 3412.121182] ? show_regs.cold.9+0x22/0x2f [ 3412.121194] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3412.121200] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3412.121210] ? watchdog+0x40/0x40 [ 3412.121214] ? hrtimer_interrupt+0x128/0x2c0 [ 3412.121218] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3412.121227] ? apic_timer_interrupt+0xf/0x20 [ 3412.121232] [ 3412.121234] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3412.121239] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3412.121245] do_raw_spin_lock+0x177/0x190 [ 3412.121249] _raw_spin_lock+0x21/0x30 [ 3412.121352] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3412.121673] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3412.122051] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3412.122386] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3412.122681] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3412.123041] ? woken_wake_function+0x30/0x30 [ 3412.123051] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3412.126147] ? do_wait_intr+0xf0/0xf0 [ 3412.126167] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3412.126504] mdc_reint+0x2d/0x110 [mdc] [ 3412.126553] mdc_unlink+0x658/0x870 [mdc] [ 3412.126580] lmv_unlink+0x34d/0xbd0 [lmv] [ 3412.126616] ll_unlink+0x24d/0x7f0 [lustre] [ 3412.126733] vfs_unlink+0x202/0x330 [ 3412.126742] do_unlinkat+0x2fa/0x460 [ 3412.126747] __x64_sys_unlink+0x37/0x50 [ 3412.126751] do_syscall_64+0xc1/0x440 [ 3412.126758] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3412.126767] RIP: 0033:0x7f83e5f5c10b [ 3412.126774] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3412.126778] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3412.126784] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3412.126786] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3412.126788] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3412.126791] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3412.126793] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3412.691540] kthread+0x1d1/0x200 [ 3412.695607] ? set_kthread_struct+0x70/0x70 [ 3412.698504] ret_from_fork+0x1f/0x30 [ 3424.061398] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 3424.067749] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3424.132551] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3424.144693] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3424.154775] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3424.157111] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3424.165468] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3424.169716] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3424.173891] RDX: 00000000000045a4 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3424.177538] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3424.181201] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3424.186353] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3424.195178] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3424.208453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3424.216409] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3424.220696] Call Trace: [ 3424.223207] [ 3424.226010] ? show_regs.cold.9+0x22/0x2f [ 3424.229302] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3424.233667] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3424.238585] ? watchdog+0x40/0x40 [ 3424.240550] ? hrtimer_interrupt+0x128/0x2c0 [ 3424.247377] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3424.250293] ? apic_timer_interrupt+0xf/0x20 [ 3424.254539] [ 3424.256872] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3424.260433] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3424.265666] do_raw_spin_lock+0x177/0x190 [ 3424.269589] _raw_spin_lock+0x21/0x30 [ 3424.272395] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3424.277408] obd_mod_free+0x46/0x140 [ptlrpc] [ 3424.280154] mdc_commit_open+0x91/0xb0 [mdc] [ 3424.282340] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3424.290527] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3424.298360] after_reply+0xe2c/0x1780 [ptlrpc] [ 3424.301133] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3424.307264] ? __next_timer_interrupt+0x160/0x160 [ 3424.316210] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3424.328531] ? do_wait_intr+0xf0/0xf0 [ 3424.332846] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3424.339178] kthread+0x1d1/0x200 [ 3424.343550] ? set_kthread_struct+0x70/0x70 [ 3424.348522] ret_from_fork+0x1f/0x30 [ 3428.060370] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3428.071249] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3428.166710] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3428.184612] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3428.201258] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3428.213727] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3428.227126] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3428.281330] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3428.303627] RAX: 0000000000000911 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3428.312436] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3428.333829] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3428.349687] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3428.371386] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3428.390445] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3428.414065] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3428.428887] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3428.446215] Call Trace: [ 3428.455000] [ 3428.460981] ? show_regs.cold.9+0x22/0x2f [ 3428.462445] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3428.474376] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3428.487324] ? watchdog+0x40/0x40 [ 3428.491210] ? hrtimer_interrupt+0x128/0x2c0 [ 3428.496679] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3428.507892] ? apic_timer_interrupt+0xf/0x20 [ 3428.509482] [ 3428.510275] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3428.517268] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3428.537400] do_raw_spin_lock+0x177/0x190 [ 3428.555850] _raw_spin_lock+0x21/0x30 [ 3428.559427] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3428.570224] process_one_work+0x2c8/0x700 [ 3428.573058] worker_thread+0x296/0x6e0 [ 3428.583870] ? rescuer_thread+0x570/0x570 [ 3428.603850] kthread+0x1d1/0x200 [ 3428.607707] ? set_kthread_struct+0x70/0x70 [ 3428.621414] ret_from_fork+0x1f/0x30 [ 3428.833132] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 915s! [ 3428.837633] Showing busy workqueues and worker pools: [ 3428.840345] workqueue events: flags=0x0 [ 3428.842770] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3428.846370] pending: osc_grant_work_handler [osc], push_to_pool [ 3428.850293] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3428.854727] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3428.863256] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3428.867908] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3428.872356] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3428.876130] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3428.881041] workqueue events_long: flags=0x0 [ 3428.883662] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3428.887456] pending: lu_objects_flush [obdclass] [ 3428.890795] workqueue events_unbound: flags=0x2 [ 3428.893079] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3428.896414] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3428.899917] workqueue events_power_efficient: flags=0x80 [ 3428.902856] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3428.906757] pending: neigh_periodic_work, do_cache_clean [ 3428.909676] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3428.912904] pending: neigh_periodic_work, check_lifetime [ 3428.915924] workqueue mm_percpu_wq: flags=0x8 [ 3428.918259] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3428.921781] pending: vmstat_update, lru_add_drain_per_cpu [ 3428.925334] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3428.929120] pending: vmstat_update [ 3428.931196] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3428.934144] pending: lru_add_drain_per_cpu BAR(48) [ 3428.936469] workqueue ipv6_addrconf: flags=0x40008 [ 3428.944732] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3428.947770] pending: addrconf_verify_work [ 3428.949372] workqueue knbd0-recv: flags=0x1a [ 3428.951630] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3428.955251] in-flight: 650:recv_work [ 3428.957213] workqueue ptlrpc_pinger: flags=0x2 [ 3428.959400] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3428.962436] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3428.965701] pool 8: cpus=0-3 flags=0x4 nice=0 hung=915s workers=4 idle: 36 466 29757 [ 3428.969714] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3419s workers=2 idle: 652 [ 3440.059022] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 3440.062296] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 3440.062533] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3440.062619] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3440.062627] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3440.062630] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3440.062644] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3440.062651] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3440.062656] RAX: 000000000000550c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3440.062658] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3440.062659] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3440.062662] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3440.062664] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3440.062667] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3440.062671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3440.062674] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3440.062683] Call Trace: [ 3440.062689] [ 3440.062698] ? show_regs.cold.9+0x22/0x2f [ 3440.062708] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3440.062714] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3440.062718] ? watchdog+0x40/0x40 [ 3440.062722] ? hrtimer_interrupt+0x128/0x2c0 [ 3440.062725] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3440.062734] ? apic_timer_interrupt+0xf/0x20 [ 3440.062738] [ 3440.062740] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3440.062745] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3440.062751] do_raw_spin_lock+0x177/0x190 [ 3440.062756] _raw_spin_lock+0x21/0x30 [ 3440.062761] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3440.063025] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3440.063322] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3440.063705] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3440.064094] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3440.064503] ? woken_wake_function+0x30/0x30 [ 3440.064513] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3440.064751] kthread+0x1d1/0x200 [ 3440.064757] ? set_kthread_struct+0x70/0x70 [ 3440.064760] ret_from_fork+0x1f/0x30 [ 3440.197088] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3440.220123] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3440.225650] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3440.230041] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3440.233228] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3440.243128] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3440.247006] RAX: 0000000000001c86 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3440.250378] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3440.257592] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3440.264924] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3440.271730] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3440.277735] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3440.288194] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3440.290740] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3440.297244] Call Trace: [ 3440.301743] [ 3440.302513] ? show_regs.cold.9+0x22/0x2f [ 3440.305860] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3440.309866] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3440.313389] ? watchdog+0x40/0x40 [ 3440.316701] ? hrtimer_interrupt+0x128/0x2c0 [ 3440.322278] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3440.324402] ? apic_timer_interrupt+0xf/0x20 [ 3440.329918] [ 3440.331013] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3440.337182] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3440.343305] do_raw_spin_lock+0x177/0x190 [ 3440.345017] _raw_spin_lock+0x21/0x30 [ 3440.350229] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3440.354319] ? do_raw_spin_unlock+0x75/0x190 [ 3440.358258] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3440.364616] ? do_wait_intr+0xf0/0xf0 [ 3440.367544] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3440.372689] mdc_reint+0x2d/0x110 [mdc] [ 3440.377691] mdc_unlink+0x658/0x870 [mdc] [ 3440.379680] lmv_unlink+0x34d/0xbd0 [lmv] [ 3440.384902] ll_unlink+0x24d/0x7f0 [lustre] [ 3440.389757] vfs_unlink+0x202/0x330 [ 3440.394161] do_unlinkat+0x2fa/0x460 [ 3440.397478] __x64_sys_unlink+0x37/0x50 [ 3440.400231] do_syscall_64+0xc1/0x440 [ 3440.405619] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3440.410019] RIP: 0033:0x7f83e5f5c10b [ 3440.413185] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3440.427461] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3440.434371] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3440.441188] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3440.447778] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3440.455323] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3440.462165] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3456.065731] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3456.066017] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 3456.085592] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3456.085667] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3456.085674] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3456.085680] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3456.085958] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3456.085968] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3456.085972] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3456.085977] RAX: 0000000000007688 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3456.085979] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3456.085981] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3456.085983] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3456.085985] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3456.085988] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3456.085991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3456.085993] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3456.086002] Call Trace: [ 3456.086008] [ 3456.086013] ? show_regs.cold.9+0x22/0x2f [ 3456.086023] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3456.086029] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3456.086033] ? watchdog+0x40/0x40 [ 3456.086038] ? hrtimer_interrupt+0x128/0x2c0 [ 3456.086042] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3456.086050] ? apic_timer_interrupt+0xf/0x20 [ 3456.086055] [ 3456.086058] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3456.086062] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3456.086067] do_raw_spin_lock+0x177/0x190 [ 3456.086072] _raw_spin_lock+0x21/0x30 [ 3456.086077] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3456.086613] process_one_work+0x2c8/0x700 [ 3456.086621] worker_thread+0x296/0x6e0 [ 3456.086636] ? rescuer_thread+0x570/0x570 [ 3456.086639] kthread+0x1d1/0x200 [ 3456.086644] ? set_kthread_struct+0x70/0x70 [ 3456.086648] ret_from_fork+0x1f/0x30 [ 3456.298041] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3456.320026] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3456.325855] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3456.330486] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3456.334041] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3456.351077] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3456.361022] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3456.366160] RDX: 000000000000401c RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3456.372556] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3456.374829] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3456.377303] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3456.380257] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3456.387002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3456.390547] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3456.393844] Call Trace: [ 3456.394569] [ 3456.395126] ? show_regs.cold.9+0x22/0x2f [ 3456.396452] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3456.397974] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3456.399983] ? watchdog+0x40/0x40 [ 3456.401234] ? hrtimer_interrupt+0x128/0x2c0 [ 3456.402783] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3456.404796] ? apic_timer_interrupt+0xf/0x20 [ 3456.406101] [ 3456.406733] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3456.407987] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3456.409870] do_raw_spin_lock+0x177/0x190 [ 3456.411129] _raw_spin_lock+0x21/0x30 [ 3456.412314] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3456.414229] obd_mod_free+0x46/0x140 [ptlrpc] [ 3456.418226] mdc_commit_open+0x91/0xb0 [mdc] [ 3456.419536] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3456.423668] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3456.426556] after_reply+0xe2c/0x1780 [ptlrpc] [ 3456.429225] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3456.432828] ? __next_timer_interrupt+0x160/0x160 [ 3456.434208] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3456.435946] ? do_wait_intr+0xf0/0xf0 [ 3456.437038] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3456.438783] kthread+0x1d1/0x200 [ 3456.439723] ? set_kthread_struct+0x70/0x70 [ 3456.440961] ret_from_fork+0x1f/0x30 [ 3459.041034] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 945s! [ 3459.045481] Showing busy workqueues and worker pools: [ 3459.048461] workqueue events: flags=0x0 [ 3459.050560] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3459.054179] pending: osc_grant_work_handler [osc], push_to_pool [ 3459.057764] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3459.061481] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3459.065202] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3459.068516] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3459.073049] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3459.076670] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3459.082270] workqueue events_long: flags=0x0 [ 3459.084491] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3459.088648] pending: lu_objects_flush [obdclass] [ 3459.091888] workqueue events_unbound: flags=0x2 [ 3459.094453] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3459.098000] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3459.101408] workqueue events_power_efficient: flags=0x80 [ 3459.104282] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3459.108438] pending: neigh_periodic_work, do_cache_clean [ 3459.112384] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3459.117321] pending: neigh_periodic_work, check_lifetime [ 3459.121049] workqueue mm_percpu_wq: flags=0x8 [ 3459.123718] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3459.127689] pending: vmstat_update, lru_add_drain_per_cpu [ 3459.131560] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3459.135720] pending: vmstat_update [ 3459.138219] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3459.143212] pending: lru_add_drain_per_cpu BAR(48) [ 3459.147262] workqueue ipv6_addrconf: flags=0x40008 [ 3459.151244] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3459.155901] pending: addrconf_verify_work [ 3459.159506] workqueue knbd0-recv: flags=0x1a [ 3459.163215] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3459.167537] in-flight: 650:recv_work [ 3459.170224] workqueue ptlrpc_pinger: flags=0x2 [ 3459.172680] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3459.175874] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3459.181713] pool 8: cpus=0-3 flags=0x4 nice=0 hung=945s workers=4 idle: 36 466 29757 [ 3459.186547] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3449s workers=2 idle: 652 [ 3468.059939] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 3468.063868] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3468.065888] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 3468.093810] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3468.093823] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3468.093826] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3468.093841] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3468.093846] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3468.093851] RAX: 0000000000007385 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3468.093853] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3468.093854] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3468.093855] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3468.093857] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3468.093859] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3468.093861] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3468.093863] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3468.093870] Call Trace: [ 3468.093875] [ 3468.093880] ? show_regs.cold.9+0x22/0x2f [ 3468.093889] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3468.093895] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3468.093900] ? watchdog+0x40/0x40 [ 3468.093904] ? hrtimer_interrupt+0x128/0x2c0 [ 3468.093907] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3468.093914] ? apic_timer_interrupt+0xf/0x20 [ 3468.093918] [ 3468.093921] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3468.093925] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3468.093932] do_raw_spin_lock+0x177/0x190 [ 3468.093936] _raw_spin_lock+0x21/0x30 [ 3468.093940] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3468.094433] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3468.094566] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3468.094709] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3468.094871] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3468.103904] Modules linked in: [ 3468.107524] ? woken_wake_function+0x30/0x30 [ 3468.107543] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3468.120553] lustre(O) [ 3468.122434] kthread+0x1d1/0x200 [ 3468.122451] ? set_kthread_struct+0x70/0x70 [ 3468.122458] ret_from_fork+0x1f/0x30 [ 3468.308063] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3468.328655] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3468.334737] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3468.339069] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3468.342018] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3468.350829] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3468.354275] RAX: 0000000000006f08 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3468.356996] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3468.359708] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3468.362277] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3468.365340] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3468.368156] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3468.371513] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3468.374334] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3468.382613] Call Trace: [ 3468.383382] [ 3468.383981] ? show_regs.cold.9+0x22/0x2f [ 3468.388420] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3468.389700] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3468.393228] ? watchdog+0x40/0x40 [ 3468.395650] ? hrtimer_interrupt+0x128/0x2c0 [ 3468.397343] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3468.399195] ? apic_timer_interrupt+0xf/0x20 [ 3468.400900] [ 3468.401765] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3468.403858] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3468.407341] do_raw_spin_lock+0x177/0x190 [ 3468.409004] _raw_spin_lock+0x21/0x30 [ 3468.410494] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3468.412853] ? do_raw_spin_unlock+0x75/0x190 [ 3468.416925] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3468.419266] ? do_wait_intr+0xf0/0xf0 [ 3468.423682] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3468.426926] mdc_reint+0x2d/0x110 [mdc] [ 3468.429045] mdc_unlink+0x658/0x870 [mdc] [ 3468.431368] lmv_unlink+0x34d/0xbd0 [lmv] [ 3468.433409] ll_unlink+0x24d/0x7f0 [lustre] [ 3468.435908] vfs_unlink+0x202/0x330 [ 3468.438482] do_unlinkat+0x2fa/0x460 [ 3468.440913] __x64_sys_unlink+0x37/0x50 [ 3468.444346] do_syscall_64+0xc1/0x440 [ 3468.450611] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3468.456875] RIP: 0033:0x7f83e5f5c10b [ 3468.459411] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3468.483809] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3468.491036] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3468.494722] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3468.498430] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3468.502104] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3468.505874] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3480.988044] rcu: INFO: rcu_sched self-detected stall on CPU [ 3480.991917] rcu: 0-....: (937372 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 3481.006548] (t=1500047 jiffies g=262441 q=57460) [ 3481.009698] rcu: rcu_sched kthread starved for 969237 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 3481.019764] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3481.029234] rcu: RCU grace-period kthread stack dump: [ 3481.036689] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3481.046332] Call Trace: [ 3481.048101] __schedule+0x351/0xcb0 [ 3481.050756] schedule+0xc0/0x180 [ 3481.055666] schedule_timeout+0xb4/0x190 [ 3481.057811] ? do_raw_spin_unlock+0x75/0x190 [ 3481.063728] ? __next_timer_interrupt+0x160/0x160 [ 3481.065721] ? prepare_to_swait_event+0xf3/0x140 [ 3481.071038] rcu_gp_kthread+0x8dd/0x15b0 [ 3481.075249] ? force_qs_rnp+0x2c0/0x2c0 [ 3481.082086] kthread+0x1d1/0x200 [ 3481.083746] ? set_kthread_struct+0x70/0x70 [ 3481.087770] ret_from_fork+0x1f/0x30 [ 3481.091044] rcu: Stack dump where RCU GP kthread last ran: [ 3481.093591] NMI backtrace for cpu 0 [ 3481.095274] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3481.100051] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3481.102528] Call Trace: [ 3481.103301] [ 3481.104045] ? dump_stack+0xbb/0x10e [ 3481.105270] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3481.107047] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3481.108786] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3481.110992] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3481.113443] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3481.115809] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 3481.118299] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 3481.120585] ? update_process_times+0x6d/0xc0 [ 3481.122336] ? tick_sched_handle.isra.18+0x27/0x90 [ 3481.124240] ? tick_sched_timer+0x53/0xb0 [ 3481.126033] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3481.128076] ? tick_sched_handle.isra.18+0x90/0x90 [ 3481.130179] ? hrtimer_interrupt+0x128/0x2c0 [ 3481.132150] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3481.134382] ? apic_timer_interrupt+0xf/0x20 [ 3481.138164] [ 3481.140259] ? native_safe_halt+0x2e/0x50 [ 3481.142283] ? kvm_wait+0xcd/0xe0 [ 3481.144113] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3481.146859] ? do_raw_spin_lock+0x177/0x190 [ 3481.149278] ? _raw_spin_lock+0x21/0x30 [ 3481.151135] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3481.153763] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3481.155782] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3481.157820] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3481.160276] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3481.163274] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3481.166192] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3481.169654] ? __next_timer_interrupt+0x160/0x160 [ 3481.171799] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3481.174065] ? do_wait_intr+0xf0/0xf0 [ 3481.176528] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3481.179529] ? kthread+0x1d1/0x200 [ 3481.181140] ? set_kthread_struct+0x70/0x70 [ 3481.183200] ? ret_from_fork+0x1f/0x30 [ 3481.184836] NMI backtrace for cpu 0 [ 3481.186830] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3481.192672] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3481.196666] Call Trace: [ 3481.197744] [ 3481.198616] ? dump_stack+0xbb/0x10e [ 3481.200252] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3481.202500] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3481.204521] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3481.206714] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3481.209087] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3481.211161] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3481.213778] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3481.217690] ? update_process_times+0x6d/0xc0 [ 3481.220511] ? tick_sched_handle.isra.18+0x27/0x90 [ 3481.224541] ? tick_sched_timer+0x53/0xb0 [ 3481.226597] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3481.229721] ? tick_sched_handle.isra.18+0x90/0x90 [ 3481.231737] ? hrtimer_interrupt+0x128/0x2c0 [ 3481.233819] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3481.236156] ? apic_timer_interrupt+0xf/0x20 [ 3481.238305] [ 3481.239346] ? native_safe_halt+0x2e/0x50 [ 3481.241842] ? kvm_wait+0xcd/0xe0 [ 3481.243335] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3481.245797] ? do_raw_spin_lock+0x177/0x190 [ 3481.247726] ? _raw_spin_lock+0x21/0x30 [ 3481.251359] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3481.253834] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3481.257202] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3481.258817] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3481.264723] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3481.273438] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3481.277106] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3481.281787] ? __next_timer_interrupt+0x160/0x160 [ 3481.284082] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3481.286542] ? do_wait_intr+0xf0/0xf0 [ 3481.288280] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3481.290798] ? kthread+0x1d1/0x200 [ 3481.292716] ? set_kthread_struct+0x70/0x70 [ 3481.295013] ? ret_from_fork+0x1f/0x30 [ 3484.063201] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3484.069719] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3484.124582] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3484.131276] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3484.136275] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3484.142203] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3484.149364] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3484.164611] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3484.171984] RAX: 0000000000007b90 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3484.176679] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3484.180955] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3484.197335] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3484.201870] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3484.205904] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3484.213679] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3484.215826] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3484.218163] Call Trace: [ 3484.219100] [ 3484.219762] ? show_regs.cold.9+0x22/0x2f [ 3484.220955] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3484.223170] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3484.224694] ? watchdog+0x40/0x40 [ 3484.225773] ? hrtimer_interrupt+0x128/0x2c0 [ 3484.227146] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3484.228621] ? apic_timer_interrupt+0xf/0x20 [ 3484.229907] [ 3484.237245] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3484.241220] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3484.246595] do_raw_spin_lock+0x177/0x190 [ 3484.249259] _raw_spin_lock+0x21/0x30 [ 3484.252574] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3484.257023] process_one_work+0x2c8/0x700 [ 3484.261309] worker_thread+0x296/0x6e0 [ 3484.262885] ? rescuer_thread+0x570/0x570 [ 3484.266328] kthread+0x1d1/0x200 [ 3484.268512] ? set_kthread_struct+0x70/0x70 [ 3484.275232] ret_from_fork+0x1f/0x30 [ 3489.249039] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 976s! [ 3489.260792] Showing busy workqueues and worker pools: [ 3489.265726] workqueue events: flags=0x0 [ 3489.274731] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3489.286921] pending: osc_grant_work_handler [osc], push_to_pool [ 3489.294759] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3489.302545] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3489.315894] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3489.327991] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3489.336280] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3489.348825] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3489.360219] workqueue events_long: flags=0x0 [ 3489.361928] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3489.368174] pending: lu_objects_flush [obdclass] [ 3489.371089] workqueue events_unbound: flags=0x2 [ 3489.373159] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3489.376179] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3489.378977] workqueue events_power_efficient: flags=0x80 [ 3489.385373] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3489.391022] pending: neigh_periodic_work, do_cache_clean [ 3489.396394] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3489.404308] pending: neigh_periodic_work, check_lifetime [ 3489.407546] workqueue mm_percpu_wq: flags=0x8 [ 3489.410701] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3489.421284] pending: vmstat_update, lru_add_drain_per_cpu [ 3489.425168] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3489.429914] pending: vmstat_update [ 3489.432420] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3489.436050] pending: lru_add_drain_per_cpu BAR(48) [ 3489.438554] workqueue ipv6_addrconf: flags=0x40008 [ 3489.450481] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3489.457956] pending: addrconf_verify_work [ 3489.459604] workqueue knbd0-recv: flags=0x1a [ 3489.464526] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3489.474817] in-flight: 650:recv_work [ 3489.479596] workqueue ptlrpc_pinger: flags=0x2 [ 3489.485483] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3489.494118] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3489.498482] pool 8: cpus=0-3 flags=0x4 nice=0 hung=976s workers=4 idle: 36 466 29757 [ 3489.507293] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3479s workers=2 idle: 652 [ 3496.060457] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 3496.062583] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 3496.066317] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3496.066401] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3496.066409] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3496.066412] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3496.066424] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3496.066428] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3496.066432] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3496.066434] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3496.066436] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3496.066438] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3496.066440] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3496.066443] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3496.066445] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3496.066447] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3496.066454] Call Trace: [ 3496.066459] [ 3496.066464] ? show_regs.cold.9+0x22/0x2f [ 3496.066473] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3496.066480] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3496.066484] ? watchdog+0x40/0x40 [ 3496.066488] ? hrtimer_interrupt+0x128/0x2c0 [ 3496.066491] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3496.066498] ? apic_timer_interrupt+0xf/0x20 [ 3496.066502] [ 3496.066505] ? native_safe_halt+0x2e/0x50 [ 3496.066509] kvm_wait+0xcd/0xe0 [ 3496.066517] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3496.066527] do_raw_spin_lock+0x177/0x190 [ 3496.066532] _raw_spin_lock+0x21/0x30 [ 3496.066536] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3496.066802] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3496.072361] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr [ 3496.097675] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3496.102695] intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3496.102738] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3496.102744] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3496.102747] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3496.102765] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3496.102769] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3496.102774] RAX: 0000000000006c3b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3496.102776] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3496.102778] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3496.102780] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3496.102782] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3496.102785] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3496.102787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3496.102789] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3496.102797] Call Trace: [ 3496.102802] [ 3496.102807] ? show_regs.cold.9+0x22/0x2f [ 3496.102816] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3496.102823] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3496.102832] ? watchdog+0x40/0x40 [ 3496.102836] ? hrtimer_interrupt+0x128/0x2c0 [ 3496.102840] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3496.102852] ? apic_timer_interrupt+0xf/0x20 [ 3496.102856] [ 3496.102860] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3496.102868] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3496.102876] do_raw_spin_lock+0x177/0x190 [ 3496.102881] _raw_spin_lock+0x21/0x30 [ 3496.102888] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3496.103255] ? do_raw_spin_unlock+0x75/0x190 [ 3496.103262] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3496.103494] ? do_wait_intr+0xf0/0xf0 [ 3496.103502] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3496.103711] mdc_reint+0x2d/0x110 [mdc] [ 3496.103750] mdc_unlink+0x658/0x870 [mdc] [ 3496.103781] lmv_unlink+0x34d/0xbd0 [lmv] [ 3496.103816] ll_unlink+0x24d/0x7f0 [lustre] [ 3496.103928] vfs_unlink+0x202/0x330 [ 3496.103936] do_unlinkat+0x2fa/0x460 [ 3496.103941] __x64_sys_unlink+0x37/0x50 [ 3496.103944] do_syscall_64+0xc1/0x440 [ 3496.103950] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3496.103957] RIP: 0033:0x7f83e5f5c10b [ 3496.103962] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3496.103966] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3496.103970] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3496.103973] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3496.103975] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3496.103977] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3496.103979] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3496.389276] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3496.391919] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3496.394573] ? woken_wake_function+0x30/0x30 [ 3496.396321] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3496.399590] kthread+0x1d1/0x200 [ 3496.400964] ? set_kthread_struct+0x70/0x70 [ 3496.402799] ret_from_fork+0x1f/0x30 [ 3512.063333] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3512.077806] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3512.163467] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3512.180516] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3512.185433] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3512.189543] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3512.194203] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3512.211334] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3512.217542] RAX: 0000000000005a87 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3512.222499] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3512.230441] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3512.239558] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3512.243746] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3512.252342] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3512.260694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3512.269264] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3512.278015] Call Trace: [ 3512.281303] [ 3512.284906] ? show_regs.cold.9+0x22/0x2f [ 3512.289350] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3512.292435] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3512.294961] ? watchdog+0x40/0x40 [ 3512.297548] ? hrtimer_interrupt+0x128/0x2c0 [ 3512.300391] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3512.307057] ? apic_timer_interrupt+0xf/0x20 [ 3512.311162] [ 3512.311763] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3512.317403] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3512.321574] do_raw_spin_lock+0x177/0x190 [ 3512.325561] _raw_spin_lock+0x21/0x30 [ 3512.333963] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3512.344177] process_one_work+0x2c8/0x700 [ 3512.347565] worker_thread+0x296/0x6e0 [ 3512.352479] ? rescuer_thread+0x570/0x570 [ 3512.356648] kthread+0x1d1/0x200 [ 3512.360386] ? set_kthread_struct+0x70/0x70 [ 3512.365741] ret_from_fork+0x1f/0x30 [ 3516.056108] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 3516.059663] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3516.080361] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3516.084887] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3516.088229] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3516.090553] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3516.098066] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3516.101072] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3516.104722] RDX: 0000000000004b08 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3516.107790] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3516.111296] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3516.114758] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3516.117884] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3516.121245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3516.123273] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3516.126522] Call Trace: [ 3516.127708] [ 3516.128457] ? show_regs.cold.9+0x22/0x2f [ 3516.130041] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3516.132481] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3516.134988] ? watchdog+0x40/0x40 [ 3516.136299] ? hrtimer_interrupt+0x128/0x2c0 [ 3516.138079] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3516.140012] ? apic_timer_interrupt+0xf/0x20 [ 3516.141732] [ 3516.142563] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3516.144833] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3516.147285] do_raw_spin_lock+0x177/0x190 [ 3516.149034] _raw_spin_lock+0x21/0x30 [ 3516.150943] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3516.153238] obd_mod_free+0x46/0x140 [ptlrpc] [ 3516.155446] mdc_commit_open+0x91/0xb0 [mdc] [ 3516.157157] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3516.159696] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3516.162368] after_reply+0xe2c/0x1780 [ptlrpc] [ 3516.164775] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3516.167252] ? __next_timer_interrupt+0x160/0x160 [ 3516.169251] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3516.171345] ? do_wait_intr+0xf0/0xf0 [ 3516.172922] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3516.175389] kthread+0x1d1/0x200 [ 3516.176758] ? set_kthread_struct+0x70/0x70 [ 3516.178544] ret_from_fork+0x1f/0x30 [ 3519.976272] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1006s! [ 3519.979560] Showing busy workqueues and worker pools: [ 3519.982930] workqueue events: flags=0x0 [ 3519.985345] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3519.999757] pending: osc_grant_work_handler [osc], push_to_pool [ 3520.008621] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3520.019603] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3520.028953] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3520.034969] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3520.048369] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3520.053489] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3520.061198] workqueue events_long: flags=0x0 [ 3520.063988] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3520.075588] pending: lu_objects_flush [obdclass] [ 3520.080349] workqueue events_unbound: flags=0x2 [ 3520.089922] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3520.099305] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3520.110991] workqueue events_power_efficient: flags=0x80 [ 3520.120736] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3520.128532] pending: neigh_periodic_work, do_cache_clean [ 3520.139939] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3520.145304] pending: neigh_periodic_work, check_lifetime [ 3520.151646] workqueue mm_percpu_wq: flags=0x8 [ 3520.153406] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3520.160923] pending: vmstat_update, lru_add_drain_per_cpu [ 3520.170265] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3520.176249] pending: vmstat_update [ 3520.179699] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3520.193553] pending: lru_add_drain_per_cpu BAR(48) [ 3520.200525] workqueue ipv6_addrconf: flags=0x40008 [ 3520.207481] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3520.212912] pending: addrconf_verify_work [ 3520.221984] workqueue knbd0-recv: flags=0x1a [ 3520.223536] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3520.233356] in-flight: 650:recv_work [ 3520.234892] workqueue ptlrpc_pinger: flags=0x2 [ 3520.247397] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3520.253684] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3520.260026] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1007s workers=4 idle: 36 466 29757 [ 3520.264589] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3510s workers=2 idle: 652 [ 3524.060962] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 3524.062023] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 3524.064961] Modules linked in: lustre(O) [ 3524.071804] Modules linked in: [ 3524.071807] osp(O) ofd(O) lod(O) [ 3524.075124] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [ 3524.076561] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3524.076640] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3524.076647] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3524.076649] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3524.076663] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3524.076667] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3524.076671] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3524.076673] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3524.076676] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3524.076678] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3524.076680] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3524.076683] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3524.076686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3524.076688] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3524.076759] Call Trace: [ 3524.076773] [ 3524.076778] ? show_regs.cold.9+0x22/0x2f [ 3524.080813] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3524.080875] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3524.080881] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3524.080884] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3524.080899] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3524.080903] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3524.080908] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3524.080911] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3524.080913] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3524.080914] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3524.080916] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3524.080920] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3524.080923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3524.080925] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3524.080934] Call Trace: [ 3524.080940] [ 3524.080947] ? show_regs.cold.9+0x22/0x2f [ 3524.080959] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3524.080966] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3524.080971] ? watchdog+0x40/0x40 [ 3524.080975] ? hrtimer_interrupt+0x128/0x2c0 [ 3524.080980] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3524.080987] ? apic_timer_interrupt+0xf/0x20 [ 3524.080991] [ 3524.080994] ? native_safe_halt+0x2e/0x50 [ 3524.080998] kvm_wait+0xcd/0xe0 [ 3524.081006] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3524.081013] do_raw_spin_lock+0x177/0x190 [ 3524.081018] _raw_spin_lock+0x21/0x30 [ 3524.081024] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3524.086608] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3524.109160] ? do_raw_spin_unlock+0x75/0x190 [ 3524.109185] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3524.109452] ? do_wait_intr+0xf0/0xf0 [ 3524.109461] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3524.110002] mdc_reint+0x2d/0x110 [mdc] [ 3524.110045] mdc_unlink+0x658/0x870 [mdc] [ 3524.110072] lmv_unlink+0x34d/0xbd0 [lmv] [ 3524.110102] ll_unlink+0x24d/0x7f0 [lustre] [ 3524.115995] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3524.116012] ? watchdog+0x40/0x40 [ 3524.116017] ? hrtimer_interrupt+0x128/0x2c0 [ 3524.116021] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3524.116029] ? apic_timer_interrupt+0xf/0x20 [ 3524.116034] [ 3524.116039] ? native_safe_halt+0x2e/0x50 [ 3524.116043] kvm_wait+0xcd/0xe0 [ 3524.116052] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3524.116063] do_raw_spin_lock+0x177/0x190 [ 3524.116066] _raw_spin_lock+0x21/0x30 [ 3524.116071] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3524.117451] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3524.117774] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3524.118022] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3524.124050] vfs_unlink+0x202/0x330 [ 3524.124075] do_unlinkat+0x2fa/0x460 [ 3524.124080] __x64_sys_unlink+0x37/0x50 [ 3524.124084] do_syscall_64+0xc1/0x440 [ 3524.124089] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3524.124105] RIP: 0033:0x7f83e5f5c10b [ 3524.124112] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3524.124116] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3524.124122] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3524.124125] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3524.124127] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3524.124129] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3524.124131] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3524.308306] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3524.310257] ? woken_wake_function+0x30/0x30 [ 3524.311482] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3524.313378] kthread+0x1d1/0x200 [ 3524.314383] ? set_kthread_struct+0x70/0x70 [ 3524.315649] ret_from_fork+0x1f/0x30 [ 3540.060355] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3540.069687] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3540.095057] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3540.101032] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3540.105733] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3540.111439] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3540.114496] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3540.125418] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3540.130039] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3540.134346] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3540.138189] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3540.141243] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3540.144047] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3540.147833] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3540.153041] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3540.156040] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3540.160626] Call Trace: [ 3540.162333] [ 3540.163211] ? show_regs.cold.9+0x22/0x2f [ 3540.165232] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3540.167529] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3540.169521] ? watchdog+0x40/0x40 [ 3540.171193] ? hrtimer_interrupt+0x128/0x2c0 [ 3540.172755] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3540.174883] ? apic_timer_interrupt+0xf/0x20 [ 3540.177513] [ 3540.178605] ? native_safe_halt+0x2e/0x50 [ 3540.180341] kvm_wait+0xcd/0xe0 [ 3540.181764] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3540.184240] do_raw_spin_lock+0x177/0x190 [ 3540.186349] _raw_spin_lock+0x21/0x30 [ 3540.187725] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3540.190363] process_one_work+0x2c8/0x700 [ 3540.192520] worker_thread+0x296/0x6e0 [ 3540.194156] ? rescuer_thread+0x570/0x570 [ 3540.198272] kthread+0x1d1/0x200 [ 3540.199574] ? set_kthread_struct+0x70/0x70 [ 3540.201312] ret_from_fork+0x1f/0x30 [ 3548.056101] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 3548.062451] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3548.105796] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3548.114233] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3548.121041] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3548.124141] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3548.136586] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3548.138481] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3548.145330] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3548.150995] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3548.156630] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3548.160587] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3548.165234] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3548.172821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3548.175710] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3548.184304] Call Trace: [ 3548.185066] [ 3548.186858] ? show_regs.cold.9+0x22/0x2f [ 3548.189061] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3548.191658] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3548.193980] ? watchdog+0x40/0x40 [ 3548.195719] ? hrtimer_interrupt+0x128/0x2c0 [ 3548.198351] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3548.202642] ? apic_timer_interrupt+0xf/0x20 [ 3548.205380] [ 3548.207156] ? native_safe_halt+0x2e/0x50 [ 3548.209519] kvm_wait+0xcd/0xe0 [ 3548.212279] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3548.214925] do_raw_spin_lock+0x177/0x190 [ 3548.216638] _raw_spin_lock+0x21/0x30 [ 3548.218371] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3548.221030] obd_mod_free+0x46/0x140 [ptlrpc] [ 3548.223369] mdc_commit_open+0x91/0xb0 [mdc] [ 3548.226303] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3548.228370] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3548.231284] after_reply+0xe2c/0x1780 [ptlrpc] [ 3548.233972] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3548.236624] ? __next_timer_interrupt+0x160/0x160 [ 3548.239570] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3548.241976] ? do_wait_intr+0xf0/0xf0 [ 3548.243921] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3548.246572] kthread+0x1d1/0x200 [ 3548.248581] ? set_kthread_struct+0x70/0x70 [ 3548.250453] ret_from_fork+0x1f/0x30 [ 3550.693727] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1037s! [ 3550.706268] Showing busy workqueues and worker pools: [ 3550.714185] workqueue events: flags=0x0 [ 3550.728141] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3550.736528] pending: osc_grant_work_handler [osc], push_to_pool [ 3550.742558] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3550.758036] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3550.776583] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3550.786144] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3550.800234] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3550.807557] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3550.828275] workqueue events_long: flags=0x0 [ 3550.833692] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3550.838828] pending: lu_objects_flush [obdclass] [ 3550.842043] workqueue events_unbound: flags=0x2 [ 3550.845600] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3550.853426] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3550.859973] workqueue events_power_efficient: flags=0x80 [ 3550.870232] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3550.883283] pending: neigh_periodic_work, do_cache_clean [ 3550.896038] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3550.908498] pending: neigh_periodic_work, check_lifetime [ 3550.919538] workqueue mm_percpu_wq: flags=0x8 [ 3550.924250] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3550.933350] pending: vmstat_update, lru_add_drain_per_cpu [ 3550.941765] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3550.955068] pending: vmstat_update [ 3550.960950] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3550.969269] pending: lru_add_drain_per_cpu BAR(48) [ 3550.976360] workqueue ipv6_addrconf: flags=0x40008 [ 3550.987579] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3550.998500] pending: addrconf_verify_work [ 3551.002882] workqueue knbd0-recv: flags=0x1a [ 3551.008840] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3551.018522] in-flight: 650:recv_work [ 3551.020343] workqueue ptlrpc_pinger: flags=0x2 [ 3551.022121] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3551.028151] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3551.034866] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1037s workers=4 idle: 36 466 29757 [ 3551.045264] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3541s workers=2 idle: 652 [ 3552.059122] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 3552.061955] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr [ 3552.062019] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 3552.065711] i2c_piix4 [ 3552.068836] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3552.068890] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3552.068897] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3552.068899] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3552.068910] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3552.068913] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3552.068916] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3552.068918] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3552.068920] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3552.068921] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3552.068923] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3552.068925] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3552.068927] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3552.068928] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3552.068935] Call Trace: [ 3552.068939] [ 3552.068944] ? show_regs.cold.9+0x22/0x2f [ 3552.068951] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3552.068955] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3552.068957] ? watchdog+0x40/0x40 [ 3552.068960] ? hrtimer_interrupt+0x128/0x2c0 [ 3552.068963] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3552.068967] ? apic_timer_interrupt+0xf/0x20 [ 3552.068970] [ 3552.068972] ? native_safe_halt+0x2e/0x50 [ 3552.068975] kvm_wait+0xcd/0xe0 [ 3552.068980] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3552.068984] do_raw_spin_lock+0x177/0x190 [ 3552.068987] _raw_spin_lock+0x21/0x30 [ 3552.068990] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3552.069195] ? do_raw_spin_unlock+0x75/0x190 [ 3552.069200] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3552.069393] ? do_wait_intr+0xf0/0xf0 [ 3552.069400] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3552.083127] squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3552.083151] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3552.083156] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3552.083158] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3552.083169] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3552.083173] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3552.083178] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3552.083181] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3552.083183] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3552.083185] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3552.083188] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3552.083191] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3552.083194] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3552.083196] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3552.083203] Call Trace: [ 3552.083209] [ 3552.083215] ? show_regs.cold.9+0x22/0x2f [ 3552.083224] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3552.083229] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3552.083233] ? watchdog+0x40/0x40 [ 3552.083237] ? hrtimer_interrupt+0x128/0x2c0 [ 3552.083240] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3552.083247] ? apic_timer_interrupt+0xf/0x20 [ 3552.083251] [ 3552.083254] ? native_safe_halt+0x2e/0x50 [ 3552.083257] kvm_wait+0xcd/0xe0 [ 3552.083263] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3552.083273] do_raw_spin_lock+0x177/0x190 [ 3552.083277] _raw_spin_lock+0x21/0x30 [ 3552.083280] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3552.083737] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3552.083944] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3552.084089] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3552.084261] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3552.084471] ? woken_wake_function+0x30/0x30 [ 3552.084478] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3552.084795] kthread+0x1d1/0x200 [ 3552.084799] ? set_kthread_struct+0x70/0x70 [ 3552.084802] ret_from_fork+0x1f/0x30 [ 3552.216513] mdc_reint+0x2d/0x110 [mdc] [ 3552.218013] mdc_unlink+0x658/0x870 [mdc] [ 3552.219465] lmv_unlink+0x34d/0xbd0 [lmv] [ 3552.221153] ll_unlink+0x24d/0x7f0 [lustre] [ 3552.223271] vfs_unlink+0x202/0x330 [ 3552.224815] do_unlinkat+0x2fa/0x460 [ 3552.226567] __x64_sys_unlink+0x37/0x50 [ 3552.228142] do_syscall_64+0xc1/0x440 [ 3552.229407] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3552.231206] RIP: 0033:0x7f83e5f5c10b [ 3552.232512] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3552.239914] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3552.243649] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3552.246397] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3552.248599] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3552.251407] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3552.254198] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3568.067402] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 3568.079213] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3568.117741] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3568.130934] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3568.135201] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3568.137513] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3568.140006] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3568.153642] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3568.161241] RAX: 000000000000606b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3568.164833] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3568.167985] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3568.173742] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3568.176813] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3568.184270] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3568.187664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3568.190255] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3568.194726] Call Trace: [ 3568.196000] [ 3568.196745] ? show_regs.cold.9+0x22/0x2f [ 3568.198444] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3568.200838] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3568.203888] ? watchdog+0x40/0x40 [ 3568.207723] ? hrtimer_interrupt+0x128/0x2c0 [ 3568.211662] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3568.218473] ? apic_timer_interrupt+0xf/0x20 [ 3568.221598] [ 3568.222432] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3568.226138] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3568.228025] do_raw_spin_lock+0x177/0x190 [ 3568.230502] _raw_spin_lock+0x21/0x30 [ 3568.232248] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3568.236268] process_one_work+0x2c8/0x700 [ 3568.237918] worker_thread+0x296/0x6e0 [ 3568.243330] ? rescuer_thread+0x570/0x570 [ 3568.246663] kthread+0x1d1/0x200 [ 3568.248328] ? set_kthread_struct+0x70/0x70 [ 3568.250218] ret_from_fork+0x1f/0x30 [ 3576.056025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 3576.059976] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3576.080953] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3576.086049] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3576.090590] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3576.093028] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3576.103074] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3576.106995] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3576.110880] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3576.114139] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3576.116667] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3576.119889] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3576.123349] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3576.127236] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3576.129141] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3576.132439] Call Trace: [ 3576.133540] [ 3576.134492] ? show_regs.cold.9+0x22/0x2f [ 3576.136447] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3576.138536] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3576.140543] ? watchdog+0x40/0x40 [ 3576.141863] ? hrtimer_interrupt+0x128/0x2c0 [ 3576.143526] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3576.145836] ? apic_timer_interrupt+0xf/0x20 [ 3576.148235] [ 3576.149455] ? native_safe_halt+0x2e/0x50 [ 3576.152185] kvm_wait+0xcd/0xe0 [ 3576.153684] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3576.156228] do_raw_spin_lock+0x177/0x190 [ 3576.158716] _raw_spin_lock+0x21/0x30 [ 3576.160837] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3576.163842] obd_mod_free+0x46/0x140 [ptlrpc] [ 3576.166677] mdc_commit_open+0x91/0xb0 [mdc] [ 3576.169098] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3576.173223] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3576.176442] after_reply+0xe2c/0x1780 [ptlrpc] [ 3576.179639] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3576.183327] ? __next_timer_interrupt+0x160/0x160 [ 3576.185934] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3576.189175] ? do_wait_intr+0xf0/0xf0 [ 3576.191800] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3576.194962] kthread+0x1d1/0x200 [ 3576.197355] ? set_kthread_struct+0x70/0x70 [ 3576.199973] ret_from_fork+0x1f/0x30 [ 3580.059214] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 3580.064233] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3580.065613] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 3580.096735] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3580.096750] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3580.096753] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3580.096769] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3580.096774] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3580.096781] RAX: 000000000000680c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3580.096783] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3580.096786] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3580.096789] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3580.096791] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3580.096795] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3580.096798] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3580.096800] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3580.096810] Call Trace: [ 3580.096815] [ 3580.096822] ? show_regs.cold.9+0x22/0x2f [ 3580.096831] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3580.096838] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3580.096843] ? watchdog+0x40/0x40 [ 3580.096847] ? hrtimer_interrupt+0x128/0x2c0 [ 3580.096852] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3580.096861] ? apic_timer_interrupt+0xf/0x20 [ 3580.096866] [ 3580.096869] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3580.096874] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3580.096880] do_raw_spin_lock+0x177/0x190 [ 3580.096885] _raw_spin_lock+0x21/0x30 [ 3580.096891] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3580.097297] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3580.097637] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3580.097873] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3580.103592] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3580.103671] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3580.103679] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3580.103681] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3580.103695] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3580.103700] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3580.103704] RAX: 0000000000007c11 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3580.103707] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3580.103709] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3580.103711] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3580.103713] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3580.103717] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3580.103719] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3580.103721] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3580.103730] Call Trace: [ 3580.103735] [ 3580.103740] ? show_regs.cold.9+0x22/0x2f [ 3580.103751] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3580.103757] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3580.103761] ? watchdog+0x40/0x40 [ 3580.103765] ? hrtimer_interrupt+0x128/0x2c0 [ 3580.103769] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3580.103779] ? apic_timer_interrupt+0xf/0x20 [ 3580.103783] [ 3580.103787] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3580.103791] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3580.103796] do_raw_spin_lock+0x177/0x190 [ 3580.103801] _raw_spin_lock+0x21/0x30 [ 3580.103806] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3580.104086] ? do_raw_spin_unlock+0x75/0x190 [ 3580.104092] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3580.104883] ? do_wait_intr+0xf0/0xf0 [ 3580.104891] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3580.105096] mdc_reint+0x2d/0x110 [mdc] [ 3580.105135] mdc_unlink+0x658/0x870 [mdc] [ 3580.105183] lmv_unlink+0x34d/0xbd0 [lmv] [ 3580.105213] ll_unlink+0x24d/0x7f0 [lustre] [ 3580.105341] vfs_unlink+0x202/0x330 [ 3580.105350] do_unlinkat+0x2fa/0x460 [ 3580.105354] __x64_sys_unlink+0x37/0x50 [ 3580.105358] do_syscall_64+0xc1/0x440 [ 3580.105365] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3580.105370] RIP: 0033:0x7f83e5f5c10b [ 3580.105400] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3580.105405] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3580.105409] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3580.105411] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3580.105413] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3580.105416] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3580.105418] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3580.451664] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3580.454237] ? woken_wake_function+0x30/0x30 [ 3580.456320] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3580.459199] kthread+0x1d1/0x200 [ 3580.461122] ? set_kthread_struct+0x70/0x70 [ 3580.462855] ret_from_fork+0x1f/0x30 [ 3581.409082] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1068s! [ 3581.420600] Showing busy workqueues and worker pools: [ 3581.423132] workqueue events: flags=0x0 [ 3581.425351] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3581.429780] pending: osc_grant_work_handler [osc], push_to_pool [ 3581.433301] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3581.437769] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3581.441783] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3581.446089] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3581.452149] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3581.456615] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3581.461729] workqueue events_long: flags=0x0 [ 3581.463582] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3581.466297] pending: lu_objects_flush [obdclass] [ 3581.468588] workqueue events_unbound: flags=0x2 [ 3581.470219] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3581.472738] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3581.475582] workqueue events_power_efficient: flags=0x80 [ 3581.477611] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3581.480246] pending: neigh_periodic_work, do_cache_clean [ 3581.482308] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3581.484681] pending: neigh_periodic_work, check_lifetime [ 3581.486509] workqueue mm_percpu_wq: flags=0x8 [ 3581.488016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3581.493598] pending: vmstat_update, lru_add_drain_per_cpu [ 3581.497163] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3581.501554] pending: vmstat_update [ 3581.504119] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3581.508237] pending: lru_add_drain_per_cpu BAR(48) [ 3581.514332] workqueue ipv6_addrconf: flags=0x40008 [ 3581.521803] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3581.532591] pending: addrconf_verify_work [ 3581.535150] workqueue knbd0-recv: flags=0x1a [ 3581.538798] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3581.551342] in-flight: 650:recv_work [ 3581.554736] workqueue ptlrpc_pinger: flags=0x2 [ 3581.557866] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3581.561458] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3581.566325] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1068s workers=4 idle: 36 466 29757 [ 3581.578293] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3571s workers=2 idle: 652 [ 3596.060033] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 3596.064221] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3596.103337] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3596.108759] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3596.115197] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3596.121222] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3596.123081] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3596.132918] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3596.139466] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3596.145883] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3596.152575] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3596.157433] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3596.162851] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3596.169866] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3596.175061] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3596.176875] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3596.183479] Call Trace: [ 3596.185726] [ 3596.187889] ? show_regs.cold.9+0x22/0x2f [ 3596.191276] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3596.195181] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3596.198972] ? watchdog+0x40/0x40 [ 3596.201063] ? hrtimer_interrupt+0x128/0x2c0 [ 3596.204608] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3596.207901] ? apic_timer_interrupt+0xf/0x20 [ 3596.209801] [ 3596.212433] ? native_safe_halt+0x2e/0x50 [ 3596.214610] kvm_wait+0xcd/0xe0 [ 3596.217462] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3596.221537] do_raw_spin_lock+0x177/0x190 [ 3596.229838] _raw_spin_lock+0x21/0x30 [ 3596.237275] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3596.242026] process_one_work+0x2c8/0x700 [ 3596.246331] worker_thread+0x296/0x6e0 [ 3596.248974] ? rescuer_thread+0x570/0x570 [ 3596.251489] kthread+0x1d1/0x200 [ 3596.252984] ? set_kthread_struct+0x70/0x70 [ 3596.255645] ret_from_fork+0x1f/0x30 [ 3608.056715] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 3608.059010] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 3608.062029] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 3608.062041] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3608.062112] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3608.062117] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3608.062120] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3608.062144] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3608.062149] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3608.062153] RAX: 0000000000001989 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3608.062155] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3608.062157] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3608.062159] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3608.062161] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3608.062164] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3608.062167] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3608.062169] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3608.062356] Call Trace: [ 3608.062362] [ 3608.062368] ? show_regs.cold.9+0x22/0x2f [ 3608.062377] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3608.062383] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3608.062387] ? watchdog+0x40/0x40 [ 3608.062391] ? hrtimer_interrupt+0x128/0x2c0 [ 3608.062395] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3608.062402] ? apic_timer_interrupt+0xf/0x20 [ 3608.062407] [ 3608.062409] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3608.062415] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3608.062426] do_raw_spin_lock+0x177/0x190 [ 3608.062430] _raw_spin_lock+0x21/0x30 [ 3608.062436] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3608.062898] ? do_raw_spin_unlock+0x75/0x190 [ 3608.062904] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3608.064152] ? do_wait_intr+0xf0/0xf0 [ 3608.064161] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3608.064529] mdc_reint+0x2d/0x110 [mdc] [ 3608.064837] mdc_unlink+0x658/0x870 [mdc] [ 3608.064866] lmv_unlink+0x34d/0xbd0 [lmv] [ 3608.064901] ll_unlink+0x24d/0x7f0 [lustre] [ 3608.065113] vfs_unlink+0x202/0x330 [ 3608.065121] do_unlinkat+0x2fa/0x460 [ 3608.065126] __x64_sys_unlink+0x37/0x50 [ 3608.065130] do_syscall_64+0xc1/0x440 [ 3608.065269] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3608.065277] RIP: 0033:0x7f83e5f5c10b [ 3608.065286] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3608.065289] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3608.065295] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3608.065297] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3608.065300] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3608.065302] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3608.065304] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3608.065383] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3608.065453] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3608.065458] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3608.065461] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3608.065471] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3608.065474] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3608.065479] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3608.065481] RDX: 0000000000000547 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3608.065484] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3608.065486] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3608.065488] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3608.065490] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3608.065493] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3608.065495] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3608.065503] Call Trace: [ 3608.065507] [ 3608.065510] ? show_regs.cold.9+0x22/0x2f [ 3608.065518] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3608.065523] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3608.065528] ? watchdog+0x40/0x40 [ 3608.065531] ? hrtimer_interrupt+0x128/0x2c0 [ 3608.065535] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3608.065540] ? apic_timer_interrupt+0xf/0x20 [ 3608.065545] [ 3608.065548] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3608.065553] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3608.065557] do_raw_spin_lock+0x177/0x190 [ 3608.065561] _raw_spin_lock+0x21/0x30 [ 3608.065566] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3608.066989] obd_mod_free+0x46/0x140 [ptlrpc] [ 3608.068051] mdc_commit_open+0x91/0xb0 [mdc] [ 3608.068087] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3608.068702] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3608.068900] after_reply+0xe2c/0x1780 [ptlrpc] [ 3608.069104] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3608.069309] ? __next_timer_interrupt+0x160/0x160 [ 3608.069321] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3608.069522] ? do_wait_intr+0xf0/0xf0 [ 3608.069532] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3608.069733] kthread+0x1d1/0x200 [ 3608.069739] ? set_kthread_struct+0x70/0x70 [ 3608.069742] ret_from_fork+0x1f/0x30 [ 3608.584974] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3608.603608] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3608.607977] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3608.611659] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 3608.614189] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 3608.622706] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3608.626562] RAX: 00000000000058b3 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3608.629244] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3608.632717] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3608.635545] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3608.638075] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3608.644556] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3608.651692] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3608.656762] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3608.662757] Call Trace: [ 3608.665254] [ 3608.666621] ? show_regs.cold.9+0x22/0x2f [ 3608.671045] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3608.680932] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3608.685889] ? watchdog+0x40/0x40 [ 3608.689345] ? hrtimer_interrupt+0x128/0x2c0 [ 3608.693447] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3608.697866] ? apic_timer_interrupt+0xf/0x20 [ 3608.703235] [ 3608.706937] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 3608.715389] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3608.723496] do_raw_spin_lock+0x177/0x190 [ 3608.727627] _raw_spin_lock+0x21/0x30 [ 3608.732537] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3608.740449] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3608.752906] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3608.768418] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3608.782499] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3608.785457] ? woken_wake_function+0x30/0x30 [ 3608.796773] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3608.813254] kthread+0x1d1/0x200 [ 3608.819206] ? set_kthread_struct+0x70/0x70 [ 3608.823536] ret_from_fork+0x1f/0x30 [ 3611.616096] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1098s! [ 3611.622704] Showing busy workqueues and worker pools: [ 3611.629180] workqueue events: flags=0x0 [ 3611.633977] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3611.641379] pending: osc_grant_work_handler [osc], push_to_pool [ 3611.649827] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3611.660824] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3611.666694] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3611.670150] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3611.673728] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3611.677443] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3611.681180] workqueue events_long: flags=0x0 [ 3611.683664] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3611.686522] pending: lu_objects_flush [obdclass] [ 3611.689405] workqueue events_unbound: flags=0x2 [ 3611.691332] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3611.696471] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3611.699466] workqueue events_power_efficient: flags=0x80 [ 3611.701822] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3611.704601] pending: neigh_periodic_work, do_cache_clean [ 3611.714379] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3611.717715] pending: neigh_periodic_work, check_lifetime [ 3611.724472] workqueue mm_percpu_wq: flags=0x8 [ 3611.727689] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3611.735685] pending: vmstat_update, lru_add_drain_per_cpu [ 3611.740511] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3611.744815] pending: vmstat_update [ 3611.746071] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3611.751618] pending: lru_add_drain_per_cpu BAR(48) [ 3611.759898] workqueue ipv6_addrconf: flags=0x40008 [ 3611.761590] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3611.763846] pending: addrconf_verify_work [ 3611.768073] workqueue knbd0-recv: flags=0x1a [ 3611.769874] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3611.772827] in-flight: 650:recv_work [ 3611.781015] workqueue ptlrpc_pinger: flags=0x2 [ 3611.782599] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3611.786401] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3611.795801] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1098s workers=4 idle: 36 466 29757 [ 3611.801489] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3601s workers=2 idle: 652 [ 3624.062701] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 3624.072132] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3624.112038] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3624.123714] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3624.138730] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3624.146864] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3624.149886] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3624.169880] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3624.176979] RAX: 0000000000007d93 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3624.182925] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3624.187742] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3624.195665] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3624.200344] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3624.205685] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3624.208941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3624.215493] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3624.219406] Call Trace: [ 3624.227122] [ 3624.230650] ? show_regs.cold.9+0x22/0x2f [ 3624.233854] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3624.238804] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3624.240851] ? watchdog+0x40/0x40 [ 3624.242309] ? hrtimer_interrupt+0x128/0x2c0 [ 3624.243721] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3624.245766] ? apic_timer_interrupt+0xf/0x20 [ 3624.248013] [ 3624.249231] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3624.254749] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3624.258874] do_raw_spin_lock+0x177/0x190 [ 3624.265338] _raw_spin_lock+0x21/0x30 [ 3624.266754] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3624.275110] process_one_work+0x2c8/0x700 [ 3624.276547] worker_thread+0x296/0x6e0 [ 3624.277711] ? rescuer_thread+0x570/0x570 [ 3624.280946] kthread+0x1d1/0x200 [ 3624.281815] ? set_kthread_struct+0x70/0x70 [ 3624.288938] ret_from_fork+0x1f/0x30 [ 3636.062042] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 3636.063061] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 3636.069399] Modules linked in: lustre(O) [ 3636.073311] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3636.073386] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3636.073393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3636.073396] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3636.073409] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3636.073413] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3636.073417] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3636.073419] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3636.073421] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3636.073424] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3636.073426] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3636.073430] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3636.073433] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3636.073435] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3636.073443] Call Trace: [ 3636.073450] [ 3636.073456] ? show_regs.cold.9+0x22/0x2f [ 3636.073465] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3636.073600] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3636.073603] ? watchdog+0x40/0x40 [ 3636.073607] ? hrtimer_interrupt+0x128/0x2c0 [ 3636.073612] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3636.073619] ? apic_timer_interrupt+0xf/0x20 [ 3636.073623] [ 3636.073627] ? native_safe_halt+0x2e/0x50 [ 3636.073630] kvm_wait+0xcd/0xe0 [ 3636.073638] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3636.073645] do_raw_spin_lock+0x177/0x190 [ 3636.073649] _raw_spin_lock+0x21/0x30 [ 3636.073653] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3636.077964] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 3636.077977] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3636.078058] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3636.078063] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3636.078066] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3636.078079] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3636.078084] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3636.078089] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3636.078091] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3636.078094] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3636.078096] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3636.078098] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3636.078101] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3636.078104] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3636.078106] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3636.078117] Call Trace: [ 3636.078121] [ 3636.078127] ? show_regs.cold.9+0x22/0x2f [ 3636.078137] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3636.078143] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3636.078148] ? watchdog+0x40/0x40 [ 3636.078153] ? hrtimer_interrupt+0x128/0x2c0 [ 3636.078157] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3636.078165] ? apic_timer_interrupt+0xf/0x20 [ 3636.078170] [ 3636.078187] ? native_safe_halt+0x2e/0x50 [ 3636.078192] kvm_wait+0xcd/0xe0 [ 3636.078199] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3636.078206] do_raw_spin_lock+0x177/0x190 [ 3636.078211] _raw_spin_lock+0x21/0x30 [ 3636.078216] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3636.078483] obd_mod_free+0x46/0x140 [ptlrpc] [ 3636.082882] osp(O) [ 3636.083878] mdc_commit_open+0x91/0xb0 [mdc] [ 3636.083924] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3636.084187] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3636.084491] after_reply+0xe2c/0x1780 [ptlrpc] [ 3636.084706] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3636.084903] ? __next_timer_interrupt+0x160/0x160 [ 3636.084914] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3636.085126] ? do_wait_intr+0xf0/0xf0 [ 3636.085135] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3636.085339] kthread+0x1d1/0x200 [ 3636.085560] ? set_kthread_struct+0x70/0x70 [ 3636.085567] ret_from_fork+0x1f/0x30 [ 3636.110955] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3636.111215] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3636.111426] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3636.122792] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3636.122873] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3636.122879] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3636.123290] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3636.123303] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3636.123307] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3636.123312] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3636.123314] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3636.123316] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3636.123318] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3636.123321] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3636.123324] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3636.123327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3636.123329] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3636.123338] Call Trace: [ 3636.123343] [ 3636.123349] ? show_regs.cold.9+0x22/0x2f [ 3636.123358] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3636.123366] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3636.123370] ? watchdog+0x40/0x40 [ 3636.123375] ? hrtimer_interrupt+0x128/0x2c0 [ 3636.123379] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3636.123387] ? apic_timer_interrupt+0xf/0x20 [ 3636.123392] [ 3636.123394] ? native_safe_halt+0x2e/0x50 [ 3636.123398] kvm_wait+0xcd/0xe0 [ 3636.123404] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3636.123412] do_raw_spin_lock+0x177/0x190 [ 3636.123415] _raw_spin_lock+0x21/0x30 [ 3636.123421] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3636.123826] ? do_raw_spin_unlock+0x75/0x190 [ 3636.129800] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3636.130812] ? woken_wake_function+0x30/0x30 [ 3636.130822] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3636.131406] kthread+0x1d1/0x200 [ 3636.131412] ? set_kthread_struct+0x70/0x70 [ 3636.131416] ret_from_fork+0x1f/0x30 [ 3636.762529] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3636.766870] ? do_wait_intr+0xf0/0xf0 [ 3636.769753] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3636.777112] mdc_reint+0x2d/0x110 [mdc] [ 3636.782213] mdc_unlink+0x658/0x870 [mdc] [ 3636.790408] lmv_unlink+0x34d/0xbd0 [lmv] [ 3636.794803] ll_unlink+0x24d/0x7f0 [lustre] [ 3636.796645] vfs_unlink+0x202/0x330 [ 3636.803570] do_unlinkat+0x2fa/0x460 [ 3636.808734] __x64_sys_unlink+0x37/0x50 [ 3636.811590] do_syscall_64+0xc1/0x440 [ 3636.816273] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3636.818690] RIP: 0033:0x7f83e5f5c10b [ 3636.820502] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3636.829811] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3636.835320] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3636.840390] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3636.843960] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3636.847707] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3636.853377] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3641.825040] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1128s! [ 3641.829696] Showing busy workqueues and worker pools: [ 3641.832547] workqueue events: flags=0x0 [ 3641.834812] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3641.838608] pending: osc_grant_work_handler [osc], push_to_pool [ 3641.842083] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3641.845607] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3641.848669] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3641.851569] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3641.855686] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3641.860009] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3641.864698] workqueue events_long: flags=0x0 [ 3641.866736] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3641.870099] pending: lu_objects_flush [obdclass] [ 3641.872285] workqueue events_unbound: flags=0x2 [ 3641.875474] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3641.878431] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3641.880971] workqueue events_power_efficient: flags=0x80 [ 3641.883654] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3641.887574] pending: neigh_periodic_work, do_cache_clean [ 3641.890978] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3641.895580] pending: neigh_periodic_work, check_lifetime [ 3641.898527] workqueue mm_percpu_wq: flags=0x8 [ 3641.900819] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3641.904098] pending: vmstat_update, lru_add_drain_per_cpu [ 3641.916144] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3641.922302] pending: vmstat_update [ 3641.924035] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3641.929923] pending: lru_add_drain_per_cpu BAR(48) [ 3641.936136] workqueue ipv6_addrconf: flags=0x40008 [ 3641.938211] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3641.944756] pending: addrconf_verify_work [ 3641.948929] workqueue knbd0-recv: flags=0x1a [ 3641.955532] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3641.958216] in-flight: 650:recv_work [ 3641.965717] workqueue ptlrpc_pinger: flags=0x2 [ 3641.972262] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3641.974808] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3641.980777] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1128s workers=4 idle: 36 466 29757 [ 3641.992356] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3632s workers=2 idle: 652 [ 3652.060401] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 3652.064181] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3652.088261] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3652.093367] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3652.097002] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3652.100441] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3652.102188] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3652.110171] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3652.113416] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3652.116651] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3652.119631] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3652.123035] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3652.126936] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3652.130589] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3652.134490] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3652.136783] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3652.140093] Call Trace: [ 3652.141612] [ 3652.142351] ? show_regs.cold.9+0x22/0x2f [ 3652.144167] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3652.146703] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3652.149101] ? watchdog+0x40/0x40 [ 3652.151325] ? hrtimer_interrupt+0x128/0x2c0 [ 3652.153454] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3652.155595] ? apic_timer_interrupt+0xf/0x20 [ 3652.157732] [ 3652.158743] ? native_safe_halt+0x2e/0x50 [ 3652.160664] kvm_wait+0xcd/0xe0 [ 3652.162171] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3652.165007] do_raw_spin_lock+0x177/0x190 [ 3652.167169] _raw_spin_lock+0x21/0x30 [ 3652.168904] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3652.172416] process_one_work+0x2c8/0x700 [ 3652.174607] worker_thread+0x296/0x6e0 [ 3652.176925] ? rescuer_thread+0x570/0x570 [ 3652.180219] kthread+0x1d1/0x200 [ 3652.182506] ? set_kthread_struct+0x70/0x70 [ 3652.186825] ret_from_fork+0x1f/0x30 [ 3660.992972] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 3661.003834] rcu: 0-....: (1053638 ticks this GP) idle=30a/1/0x4000000000000000 softirq=128519/128519 fqs=126395 [ 3661.020632] (detected by 1, t=1680033 jiffies, g=262441, q=57491) [ 3661.029144] Sending NMI from CPU 1 to CPUs 0: [ 3661.037632] NMI backtrace for cpu 0 [ 3661.037639] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3661.037641] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3661.037642] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3661.037644] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3661.037647] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 [ 3661.037650] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3661.037652] RDX: 0000000000006840 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3661.037653] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3661.037654] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3661.037656] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3661.037657] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3661.037658] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3661.037660] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3661.037661] Call Trace: [ 3661.037662] [ 3661.037663] ? show_regs.cold.9+0x22/0x2f [ 3661.037664] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3661.037666] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3661.037667] ? nmi_handle+0xbd/0x210 [ 3661.037668] ? default_do_nmi+0x4b/0x1b0 [ 3661.037669] ? do_nmi+0x280/0x380 [ 3661.037670] ? end_repeat_nmi+0x16/0x1f [ 3661.037671] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3661.037672] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3661.037673] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3661.037674] [ 3661.037675] do_raw_spin_lock+0x177/0x190 [ 3661.037676] _raw_spin_lock+0x21/0x30 [ 3661.037677] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3661.037677] obd_mod_free+0x46/0x140 [ptlrpc] [ 3661.037678] mdc_commit_open+0x91/0xb0 [mdc] [ 3661.037679] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3661.037680] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3661.037681] after_reply+0xe2c/0x1780 [ptlrpc] [ 3661.037682] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3661.037683] ? __next_timer_interrupt+0x160/0x160 [ 3661.037684] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3661.037685] ? do_wait_intr+0xf0/0xf0 [ 3661.037686] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3661.037687] kthread+0x1d1/0x200 [ 3661.037688] ? set_kthread_struct+0x70/0x70 [ 3661.037689] ret_from_fork+0x1f/0x30 [ 3661.043824] rcu: rcu_sched kthread starved for 1149220 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 3661.252830] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3661.257614] rcu: RCU grace-period kthread stack dump: [ 3661.262202] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3661.267877] Call Trace: [ 3661.268823] __schedule+0x351/0xcb0 [ 3661.273280] schedule+0xc0/0x180 [ 3661.275028] schedule_timeout+0xb4/0x190 [ 3661.280397] ? do_raw_spin_unlock+0x75/0x190 [ 3661.284015] ? __next_timer_interrupt+0x160/0x160 [ 3661.289066] ? prepare_to_swait_event+0xf3/0x140 [ 3661.293249] rcu_gp_kthread+0x8dd/0x15b0 [ 3661.296776] ? force_qs_rnp+0x2c0/0x2c0 [ 3661.298494] kthread+0x1d1/0x200 [ 3661.299914] ? set_kthread_struct+0x70/0x70 [ 3661.309247] ret_from_fork+0x1f/0x30 [ 3661.316169] rcu: Stack dump where RCU GP kthread last ran: [ 3661.322817] Sending NMI from CPU 1 to CPUs 3: [ 3661.335171] NMI backtrace for cpu 3 [ 3661.335177] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3661.335179] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3661.335185] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3661.335186] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3661.335189] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 [ 3661.335191] RAX: 0000000000001f93 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3661.335193] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3661.335195] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3661.335196] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3661.335197] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3661.335199] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3661.335200] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3661.335201] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3661.335202] Call Trace: [ 3661.335203] [ 3661.335204] ? show_regs.cold.9+0x22/0x2f [ 3661.335205] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3661.335206] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3661.335207] ? nmi_handle+0xbd/0x210 [ 3661.335208] ? default_do_nmi+0x4b/0x1b0 [ 3661.335209] ? do_nmi+0x280/0x380 [ 3661.335209] ? end_repeat_nmi+0x16/0x1f [ 3661.335210] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3661.335211] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3661.335221] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3661.335222] [ 3661.335223] do_raw_spin_lock+0x177/0x190 [ 3661.335224] _raw_spin_lock+0x21/0x30 [ 3661.335224] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3661.335225] ? do_raw_spin_unlock+0x75/0x190 [ 3661.335226] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3661.335227] ? do_wait_intr+0xf0/0xf0 [ 3661.335228] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3661.335229] mdc_reint+0x2d/0x110 [mdc] [ 3661.335230] mdc_unlink+0x658/0x870 [mdc] [ 3661.335231] lmv_unlink+0x34d/0xbd0 [lmv] [ 3661.335232] ll_unlink+0x24d/0x7f0 [lustre] [ 3661.335234] vfs_unlink+0x202/0x330 [ 3661.335235] do_unlinkat+0x2fa/0x460 [ 3661.335236] __x64_sys_unlink+0x37/0x50 [ 3661.335237] do_syscall_64+0xc1/0x440 [ 3661.335238] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3661.335239] RIP: 0033:0x7f83e5f5c10b [ 3661.335241] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3661.335243] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3661.335245] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3661.335246] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3661.335248] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3661.335249] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3661.335251] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3672.035030] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1158s! [ 3672.037992] Showing busy workqueues and worker pools: [ 3672.040340] workqueue events: flags=0x0 [ 3672.042097] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3672.044991] pending: osc_grant_work_handler [osc], push_to_pool [ 3672.047994] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3672.050975] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3672.054216] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3672.058473] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3672.062615] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3672.065901] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3672.070488] workqueue events_long: flags=0x0 [ 3672.072020] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3672.075084] pending: lu_objects_flush [obdclass] [ 3672.077428] workqueue events_unbound: flags=0x2 [ 3672.079558] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3672.082603] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3672.085666] workqueue events_power_efficient: flags=0x80 [ 3672.088495] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3672.091392] pending: neigh_periodic_work, do_cache_clean [ 3672.094020] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3672.096289] pending: neigh_periodic_work, check_lifetime [ 3672.107627] workqueue mm_percpu_wq: flags=0x8 [ 3672.112028] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3672.116868] pending: vmstat_update, lru_add_drain_per_cpu [ 3672.120795] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3672.124170] pending: vmstat_update [ 3672.129917] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3672.133850] pending: lru_add_drain_per_cpu BAR(48) [ 3672.136295] workqueue ipv6_addrconf: flags=0x40008 [ 3672.142413] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3672.150339] pending: addrconf_verify_work [ 3672.153364] workqueue knbd0-recv: flags=0x1a [ 3672.155633] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3672.158906] in-flight: 650:recv_work [ 3672.161711] workqueue ptlrpc_pinger: flags=0x2 [ 3672.163646] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3672.166648] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3672.169510] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1158s workers=4 idle: 36 466 29757 [ 3672.172515] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3662s workers=2 idle: 652 [ 3680.060034] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 3680.062306] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3680.106159] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3680.117394] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3680.128013] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3680.132429] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3680.135571] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3680.160757] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3680.167624] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3680.173228] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3680.176411] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3680.179288] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3680.181882] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3680.185012] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3680.188292] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3680.191781] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3680.196345] Call Trace: [ 3680.197362] [ 3680.198729] ? show_regs.cold.9+0x22/0x2f [ 3680.202226] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3680.208598] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3680.209976] ? watchdog+0x40/0x40 [ 3680.212999] ? hrtimer_interrupt+0x128/0x2c0 [ 3680.215285] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3680.218030] ? apic_timer_interrupt+0xf/0x20 [ 3680.220862] [ 3680.222000] ? native_safe_halt+0x2e/0x50 [ 3680.226796] kvm_wait+0xcd/0xe0 [ 3680.229055] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3680.231713] do_raw_spin_lock+0x177/0x190 [ 3680.233617] _raw_spin_lock+0x21/0x30 [ 3680.235021] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3680.240807] process_one_work+0x2c8/0x700 [ 3680.243633] worker_thread+0x296/0x6e0 [ 3680.246113] ? rescuer_thread+0x570/0x570 [ 3680.250298] kthread+0x1d1/0x200 [ 3680.252372] ? set_kthread_struct+0x70/0x70 [ 3680.255496] ret_from_fork+0x1f/0x30 [ 3688.065138] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 3688.066196] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 3688.068598] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3688.068678] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3688.068685] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3688.068688] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3688.068701] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3688.068705] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3688.068710] RAX: 0000000000006f13 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3688.068712] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3688.068714] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3688.068717] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3688.068719] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3688.068723] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3688.068726] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3688.068728] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3688.068736] Call Trace: [ 3688.068741] [ 3688.068747] ? show_regs.cold.9+0x22/0x2f [ 3688.068755] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3688.068762] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3688.068766] ? watchdog+0x40/0x40 [ 3688.068770] ? hrtimer_interrupt+0x128/0x2c0 [ 3688.068774] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3688.068783] ? apic_timer_interrupt+0xf/0x20 [ 3688.068787] [ 3688.068789] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3688.068794] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3688.068798] do_raw_spin_lock+0x177/0x190 [ 3688.068803] _raw_spin_lock+0x21/0x30 [ 3688.068808] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3688.069290] ? do_raw_spin_unlock+0x75/0x190 [ 3688.069296] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3688.069561] ? do_wait_intr+0xf0/0xf0 [ 3688.069569] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3688.069770] mdc_reint+0x2d/0x110 [mdc] [ 3688.069807] mdc_unlink+0x658/0x870 [mdc] [ 3688.069834] lmv_unlink+0x34d/0xbd0 [lmv] [ 3688.069863] ll_unlink+0x24d/0x7f0 [lustre] [ 3688.069977] vfs_unlink+0x202/0x330 [ 3688.069986] do_unlinkat+0x2fa/0x460 [ 3688.069991] __x64_sys_unlink+0x37/0x50 [ 3688.069993] do_syscall_64+0xc1/0x440 [ 3688.069999] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3688.070005] RIP: 0033:0x7f83e5f5c10b [ 3688.070012] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3688.070015] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3688.070020] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3688.070021] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3688.070023] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3688.070026] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3688.070028] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3688.297169] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3688.354734] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3688.364060] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3688.369065] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3688.372799] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3688.384014] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3688.397550] RAX: 000000000000298a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3688.404535] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3688.414840] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3688.424332] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3688.431022] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3688.437347] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3688.445422] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3688.450954] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3688.454694] Call Trace: [ 3688.455865] [ 3688.456950] ? show_regs.cold.9+0x22/0x2f [ 3688.459098] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3688.462963] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3688.465036] ? watchdog+0x40/0x40 [ 3688.466498] ? hrtimer_interrupt+0x128/0x2c0 [ 3688.468369] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3688.470160] ? apic_timer_interrupt+0xf/0x20 [ 3688.471786] [ 3688.472469] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3688.474278] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3688.476748] do_raw_spin_lock+0x177/0x190 [ 3688.478484] _raw_spin_lock+0x21/0x30 [ 3688.480283] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3688.483953] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3688.487079] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3688.489844] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3688.492109] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3688.496880] ? woken_wake_function+0x30/0x30 [ 3688.501010] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3688.504815] kthread+0x1d1/0x200 [ 3688.506430] ? set_kthread_struct+0x70/0x70 [ 3688.508415] ret_from_fork+0x1f/0x30 [ 3700.056368] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 3700.060227] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3700.086517] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3700.091585] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3700.095447] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3700.098079] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3700.106420] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3700.110188] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3700.113479] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3700.116775] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3700.120243] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3700.123670] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3700.127109] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3700.131254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3700.134436] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3700.138519] Call Trace: [ 3700.139924] [ 3700.141065] ? show_regs.cold.9+0x22/0x2f [ 3700.143099] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3700.145393] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3700.146795] ? watchdog+0x40/0x40 [ 3700.148184] ? hrtimer_interrupt+0x128/0x2c0 [ 3700.150055] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3700.152004] ? apic_timer_interrupt+0xf/0x20 [ 3700.154108] [ 3700.155166] ? native_safe_halt+0x2e/0x50 [ 3700.157086] kvm_wait+0xcd/0xe0 [ 3700.158925] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3700.161800] do_raw_spin_lock+0x177/0x190 [ 3700.163696] _raw_spin_lock+0x21/0x30 [ 3700.165438] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3700.168035] obd_mod_free+0x46/0x140 [ptlrpc] [ 3700.170996] mdc_commit_open+0x91/0xb0 [mdc] [ 3700.172997] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3700.175475] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3700.178365] after_reply+0xe2c/0x1780 [ptlrpc] [ 3700.180874] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3700.183528] ? __next_timer_interrupt+0x160/0x160 [ 3700.184950] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3700.186910] ? do_wait_intr+0xf0/0xf0 [ 3700.188118] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3700.190084] kthread+0x1d1/0x200 [ 3700.191163] ? set_kthread_struct+0x70/0x70 [ 3700.192649] ret_from_fork+0x1f/0x30 [ 3702.249062] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1189s! [ 3702.252722] Showing busy workqueues and worker pools: [ 3702.256309] workqueue events: flags=0x0 [ 3702.259213] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3702.271888] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 3702.285092] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3702.288061] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3702.292467] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3702.298755] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3702.304030] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3702.307983] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3702.313789] workqueue events_long: flags=0x0 [ 3702.316919] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3702.319723] pending: lu_objects_flush [obdclass] [ 3702.323281] workqueue events_unbound: flags=0x2 [ 3702.326359] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3702.331801] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3702.335283] workqueue events_power_efficient: flags=0x80 [ 3702.338707] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3702.342868] pending: neigh_periodic_work, do_cache_clean [ 3702.346417] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3702.350824] pending: neigh_periodic_work, check_lifetime [ 3702.353351] workqueue mm_percpu_wq: flags=0x8 [ 3702.356236] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3702.359911] pending: vmstat_update, lru_add_drain_per_cpu [ 3702.363135] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3702.367718] pending: vmstat_update [ 3702.369804] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3702.373604] pending: lru_add_drain_per_cpu BAR(48) [ 3702.376737] workqueue ipv6_addrconf: flags=0x40008 [ 3702.379545] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3702.383583] pending: addrconf_verify_work [ 3702.386105] workqueue knbd0-recv: flags=0x1a [ 3702.388623] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3702.391968] in-flight: 650:recv_work [ 3702.394151] workqueue ptlrpc_pinger: flags=0x2 [ 3702.396143] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3702.399552] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3702.402603] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1189s workers=4 idle: 36 466 29757 [ 3702.407634] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3692s workers=2 idle: 652 [ 3708.064059] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3708.072529] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3708.095431] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3708.101399] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3708.108058] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3708.111930] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3708.115519] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3708.128297] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3708.133345] RAX: 000000000000768f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3708.136897] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3708.140795] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3708.144406] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3708.147617] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3708.151511] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3708.155936] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3708.158117] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3708.161597] Call Trace: [ 3708.162814] [ 3708.163950] ? show_regs.cold.9+0x22/0x2f [ 3708.166301] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3708.168941] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3708.174731] ? watchdog+0x40/0x40 [ 3708.176943] ? hrtimer_interrupt+0x128/0x2c0 [ 3708.179503] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3708.181451] ? apic_timer_interrupt+0xf/0x20 [ 3708.182748] [ 3708.183404] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3708.185584] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3708.189573] do_raw_spin_lock+0x177/0x190 [ 3708.191912] _raw_spin_lock+0x21/0x30 [ 3708.194146] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3708.197927] process_one_work+0x2c8/0x700 [ 3708.207111] worker_thread+0x296/0x6e0 [ 3708.209972] ? rescuer_thread+0x570/0x570 [ 3708.214443] kthread+0x1d1/0x200 [ 3708.216423] ? set_kthread_struct+0x70/0x70 [ 3708.223115] ret_from_fork+0x1f/0x30 [ 3716.061588] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 3716.064586] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 3716.074190] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) [ 3716.079073] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3716.079162] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3716.079170] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3716.079173] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3716.079187] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3716.079191] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3716.079196] RAX: 000000000000288c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3716.079198] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3716.079200] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3716.079202] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3716.079204] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3716.079207] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3716.079210] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3716.079212] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3716.079221] Call Trace: [ 3716.079226] [ 3716.079232] ? show_regs.cold.9+0x22/0x2f [ 3716.079242] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3716.079248] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3716.079252] ? watchdog+0x40/0x40 [ 3716.079257] ? hrtimer_interrupt+0x128/0x2c0 [ 3716.079261] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3716.079270] ? apic_timer_interrupt+0xf/0x20 [ 3716.079275] [ 3716.079277] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3716.079282] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3716.079286] do_raw_spin_lock+0x177/0x190 [ 3716.079291] _raw_spin_lock+0x21/0x30 [ 3716.079297] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3716.079679] ? do_raw_spin_unlock+0x75/0x190 [ 3716.079686] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3716.080176] ? do_wait_intr+0xf0/0xf0 [ 3716.080189] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3716.080410] mdc_reint+0x2d/0x110 [mdc] [ 3716.080461] mdc_unlink+0x658/0x870 [mdc] [ 3716.080563] lmv_unlink+0x34d/0xbd0 [lmv] [ 3716.080607] ll_unlink+0x24d/0x7f0 [lustre] [ 3716.080974] vfs_unlink+0x202/0x330 [ 3716.080983] do_unlinkat+0x2fa/0x460 [ 3716.080988] __x64_sys_unlink+0x37/0x50 [ 3716.080991] do_syscall_64+0xc1/0x440 [ 3716.080997] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3716.081010] RIP: 0033:0x7f83e5f5c10b [ 3716.081018] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3716.081021] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3716.081026] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3716.081028] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3716.081030] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3716.081032] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3716.081033] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3716.431736] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3716.450950] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3716.455884] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3716.460009] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3716.462973] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3716.472255] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3716.475291] RAX: 0000000000001186 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3716.478244] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3716.481585] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3716.488465] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3716.491540] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3716.495384] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3716.500908] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3716.503879] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3716.507637] Call Trace: [ 3716.509050] [ 3716.509844] ? show_regs.cold.9+0x22/0x2f [ 3716.512522] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3716.515219] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3716.517782] ? watchdog+0x40/0x40 [ 3716.519601] ? hrtimer_interrupt+0x128/0x2c0 [ 3716.521654] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3716.524088] ? apic_timer_interrupt+0xf/0x20 [ 3716.528875] [ 3716.530077] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3716.534591] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3716.537028] do_raw_spin_lock+0x177/0x190 [ 3716.539841] _raw_spin_lock+0x21/0x30 [ 3716.543058] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3716.547073] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3716.552304] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3716.556071] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3716.560027] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3716.565991] ? woken_wake_function+0x30/0x30 [ 3716.569025] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3716.573528] kthread+0x1d1/0x200 [ 3716.576475] ? set_kthread_struct+0x70/0x70 [ 3716.579379] ret_from_fork+0x1f/0x30 [ 3728.059998] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 3728.070464] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3728.121734] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3728.128674] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3728.142831] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3728.147339] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3728.157838] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3728.170304] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3728.178664] RDX: 0000000000007d7a RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3728.186006] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3728.188053] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3728.191848] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3728.197398] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3728.208841] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3728.215469] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3728.221459] Call Trace: [ 3728.222751] [ 3728.227585] ? show_regs.cold.9+0x22/0x2f [ 3728.230098] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3728.242629] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3728.253698] ? watchdog+0x40/0x40 [ 3728.266288] ? hrtimer_interrupt+0x128/0x2c0 [ 3728.274166] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3728.290104] ? apic_timer_interrupt+0xf/0x20 [ 3728.291580] [ 3728.292285] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3728.306808] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3728.310732] do_raw_spin_lock+0x177/0x190 [ 3728.313609] _raw_spin_lock+0x21/0x30 [ 3728.315840] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3728.318617] obd_mod_free+0x46/0x140 [ptlrpc] [ 3728.321481] mdc_commit_open+0x91/0xb0 [mdc] [ 3728.324106] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3728.331231] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3728.337694] after_reply+0xe2c/0x1780 [ptlrpc] [ 3728.342283] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3728.347531] ? __next_timer_interrupt+0x160/0x160 [ 3728.350306] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3728.355935] ? do_wait_intr+0xf0/0xf0 [ 3728.357601] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3728.365450] kthread+0x1d1/0x200 [ 3728.367061] ? set_kthread_struct+0x70/0x70 [ 3728.375027] ret_from_fork+0x1f/0x30 [ 3732.448146] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1219s! [ 3732.454124] Showing busy workqueues and worker pools: [ 3732.456885] workqueue events: flags=0x0 [ 3732.459367] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3732.463874] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 3732.469194] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3732.472679] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3732.477018] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3732.481114] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3732.485254] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3732.488761] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3732.493870] workqueue events_long: flags=0x0 [ 3732.496222] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3732.500216] pending: lu_objects_flush [obdclass] [ 3732.503527] workqueue events_unbound: flags=0x2 [ 3732.505733] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3732.509537] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3732.512919] workqueue events_power_efficient: flags=0x80 [ 3732.515188] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3732.518116] pending: neigh_periodic_work, do_cache_clean [ 3732.526807] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3732.531275] pending: neigh_periodic_work, check_lifetime [ 3732.533544] workqueue mm_percpu_wq: flags=0x8 [ 3732.535561] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3732.537601] pending: vmstat_update, lru_add_drain_per_cpu [ 3732.539691] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3732.547280] pending: vmstat_update [ 3732.550338] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3732.556585] pending: lru_add_drain_per_cpu BAR(48) [ 3732.559903] workqueue ipv6_addrconf: flags=0x40008 [ 3732.562390] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3732.565269] pending: addrconf_verify_work [ 3732.566796] workqueue knbd0-recv: flags=0x1a [ 3732.568508] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3732.570569] in-flight: 650:recv_work [ 3732.572227] workqueue ptlrpc_pinger: flags=0x2 [ 3732.573831] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3732.576522] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3732.578742] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1219s workers=4 idle: 36 466 29757 [ 3732.581740] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3722s workers=2 idle: 652 [ 3736.060025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3736.064774] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3736.089820] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3736.098986] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3736.111365] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3736.116914] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3736.124034] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3736.169893] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3736.183735] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3736.200400] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3736.204652] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3736.207990] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3736.209903] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3736.222490] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3736.229061] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3736.236439] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3736.244048] Call Trace: [ 3736.248726] [ 3736.251052] ? show_regs.cold.9+0x22/0x2f [ 3736.255469] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3736.259289] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3736.263539] ? watchdog+0x40/0x40 [ 3736.264695] ? hrtimer_interrupt+0x128/0x2c0 [ 3736.266013] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3736.275641] ? apic_timer_interrupt+0xf/0x20 [ 3736.278835] [ 3736.279679] ? native_safe_halt+0x2e/0x50 [ 3736.281179] kvm_wait+0xcd/0xe0 [ 3736.283056] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3736.286740] do_raw_spin_lock+0x177/0x190 [ 3736.292122] _raw_spin_lock+0x21/0x30 [ 3736.295448] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3736.303496] process_one_work+0x2c8/0x700 [ 3736.307478] worker_thread+0x296/0x6e0 [ 3736.315405] ? rescuer_thread+0x570/0x570 [ 3736.318767] kthread+0x1d1/0x200 [ 3736.321428] ? set_kthread_struct+0x70/0x70 [ 3736.323025] ret_from_fork+0x1f/0x30 [ 3744.061510] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 3744.063778] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 3744.082366] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3744.082662] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3744.082670] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3744.082673] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3744.082687] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3744.082692] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3744.082696] RAX: 00000000000052ab RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3744.082698] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3744.082701] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3744.082703] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3744.082705] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3744.082708] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3744.082711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3744.082718] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3744.082725] Call Trace: [ 3744.082731] [ 3744.082737] ? show_regs.cold.9+0x22/0x2f [ 3744.082746] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3744.082752] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3744.082756] ? watchdog+0x40/0x40 [ 3744.082760] ? hrtimer_interrupt+0x128/0x2c0 [ 3744.082769] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3744.082778] ? apic_timer_interrupt+0xf/0x20 [ 3744.082783] [ 3744.082786] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3744.082790] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3744.082795] do_raw_spin_lock+0x177/0x190 [ 3744.082801] _raw_spin_lock+0x21/0x30 [ 3744.082806] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3744.083124] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3744.083583] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3744.084167] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3744.102434] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3744.102511] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3744.102518] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3744.102521] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3744.102535] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3744.102538] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3744.102543] RAX: 000000000000381e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3744.102544] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3744.102547] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3744.102549] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3744.102550] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3744.102553] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3744.102556] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3744.102557] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3744.102568] Call Trace: [ 3744.102574] [ 3744.102580] ? show_regs.cold.9+0x22/0x2f [ 3744.102588] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3744.102594] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3744.102598] ? watchdog+0x40/0x40 [ 3744.102602] ? hrtimer_interrupt+0x128/0x2c0 [ 3744.102606] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3744.102621] ? apic_timer_interrupt+0xf/0x20 [ 3744.102717] [ 3744.102721] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3744.102726] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3744.102736] do_raw_spin_lock+0x177/0x190 [ 3744.102741] _raw_spin_lock+0x21/0x30 [ 3744.102750] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3744.103033] ? do_raw_spin_unlock+0x75/0x190 [ 3744.103039] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3744.103357] ? do_wait_intr+0xf0/0xf0 [ 3744.103365] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3744.103568] mdc_reint+0x2d/0x110 [mdc] [ 3744.103603] mdc_unlink+0x658/0x870 [mdc] [ 3744.103765] lmv_unlink+0x34d/0xbd0 [lmv] [ 3744.103798] ll_unlink+0x24d/0x7f0 [lustre] [ 3744.104072] vfs_unlink+0x202/0x330 [ 3744.104081] do_unlinkat+0x2fa/0x460 [ 3744.104086] __x64_sys_unlink+0x37/0x50 [ 3744.104090] do_syscall_64+0xc1/0x440 [ 3744.104097] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3744.104103] RIP: 0033:0x7f83e5f5c10b [ 3744.104110] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3744.104114] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3744.104119] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3744.104121] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3744.104123] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3744.104125] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3744.104127] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3744.859306] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3744.866148] ? woken_wake_function+0x30/0x30 [ 3744.869384] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3744.888004] kthread+0x1d1/0x200 [ 3744.896213] ? set_kthread_struct+0x70/0x70 [ 3744.900971] ret_from_fork+0x1f/0x30 [ 3760.058315] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 3760.071505] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3760.156881] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3760.175783] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3760.194470] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3760.208497] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3760.243042] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3760.258810] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3760.270206] RDX: 00000000000021e5 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3760.283607] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3760.295987] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3760.307121] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3760.317645] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3760.323361] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3760.327452] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3760.335159] Call Trace: [ 3760.336150] [ 3760.336929] ? show_regs.cold.9+0x22/0x2f [ 3760.338367] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3760.352278] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3760.365448] ? watchdog+0x40/0x40 [ 3760.374207] ? hrtimer_interrupt+0x128/0x2c0 [ 3760.380050] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3760.383337] ? apic_timer_interrupt+0xf/0x20 [ 3760.392877] [ 3760.397019] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3760.406128] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3760.411460] do_raw_spin_lock+0x177/0x190 [ 3760.417167] _raw_spin_lock+0x21/0x30 [ 3760.426050] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3760.430162] obd_mod_free+0x46/0x140 [ptlrpc] [ 3760.436912] mdc_commit_open+0x91/0xb0 [mdc] [ 3760.444465] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3760.454962] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3760.472156] after_reply+0xe2c/0x1780 [ptlrpc] [ 3760.481346] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3760.491505] ? __next_timer_interrupt+0x160/0x160 [ 3760.501363] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3760.511268] ? do_wait_intr+0xf0/0xf0 [ 3760.512511] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3760.524256] kthread+0x1d1/0x200 [ 3760.527548] ? set_kthread_struct+0x70/0x70 [ 3760.536001] ret_from_fork+0x1f/0x30 [ 3762.664442] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1249s! [ 3762.679048] Showing busy workqueues and worker pools: [ 3762.683050] workqueue events: flags=0x0 [ 3762.686372] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3762.702260] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 3762.713032] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3762.724312] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3762.727894] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3762.735289] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3762.740558] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3762.746038] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3762.752238] workqueue events_long: flags=0x0 [ 3762.758040] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3762.763433] pending: lu_objects_flush [obdclass] [ 3762.767118] workqueue events_unbound: flags=0x2 [ 3762.770144] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3762.773207] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3762.776399] workqueue events_power_efficient: flags=0x80 [ 3762.782836] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3762.792974] pending: neigh_periodic_work, do_cache_clean [ 3762.797099] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3762.807555] pending: neigh_periodic_work, check_lifetime [ 3762.819262] workqueue mm_percpu_wq: flags=0x8 [ 3762.825036] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3762.834557] pending: vmstat_update, lru_add_drain_per_cpu [ 3762.840261] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3762.846619] pending: vmstat_update [ 3762.855249] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3762.863021] pending: lru_add_drain_per_cpu BAR(48) [ 3762.866495] workqueue ipv6_addrconf: flags=0x40008 [ 3762.878025] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3762.888448] pending: addrconf_verify_work [ 3762.899512] workqueue knbd0-recv: flags=0x1a [ 3762.912174] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3762.928218] in-flight: 650:recv_work [ 3762.938899] workqueue ptlrpc_pinger: flags=0x2 [ 3762.940437] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3762.947192] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3762.953561] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1249s workers=4 idle: 36 466 29757 [ 3762.966105] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3753s workers=2 idle: 652 [ 3764.064170] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3764.072315] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3764.100999] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3764.105581] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3764.110388] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3764.113816] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3764.116494] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3764.124164] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3764.128331] RAX: 0000000000006187 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3764.139850] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3764.148709] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3764.155657] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3764.161461] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3764.165914] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3764.174989] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3764.182553] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3764.189580] Call Trace: [ 3764.190603] [ 3764.192346] ? show_regs.cold.9+0x22/0x2f [ 3764.196083] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3764.201039] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3764.205019] ? watchdog+0x40/0x40 [ 3764.208279] ? hrtimer_interrupt+0x128/0x2c0 [ 3764.210605] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3764.212431] ? apic_timer_interrupt+0xf/0x20 [ 3764.216676] [ 3764.217540] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3764.224502] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3764.230413] do_raw_spin_lock+0x177/0x190 [ 3764.234300] _raw_spin_lock+0x21/0x30 [ 3764.237656] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3764.245510] process_one_work+0x2c8/0x700 [ 3764.248063] worker_thread+0x296/0x6e0 [ 3764.251698] ? rescuer_thread+0x570/0x570 [ 3764.255527] kthread+0x1d1/0x200 [ 3764.260545] ? set_kthread_struct+0x70/0x70 [ 3764.261820] ret_from_fork+0x1f/0x30 [ 3772.061138] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 3772.062064] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 3772.071774] Modules linked in: [ 3772.076533] Modules linked in: [ 3772.076535] lustre(O) osp(O) [ 3772.078066] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3772.078139] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3772.078146] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3772.078149] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3772.078295] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3772.078299] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3772.078305] RAX: 0000000000001ffb RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3772.078308] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3772.078310] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3772.078312] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3772.078314] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3772.078318] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3772.078320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3772.078323] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3772.078332] Call Trace: [ 3772.078337] [ 3772.078343] ? show_regs.cold.9+0x22/0x2f [ 3772.078352] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3772.078358] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3772.078372] ? watchdog+0x40/0x40 [ 3772.078377] ? hrtimer_interrupt+0x128/0x2c0 [ 3772.078380] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3772.078388] ? apic_timer_interrupt+0xf/0x20 [ 3772.078392] [ 3772.078395] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3772.078401] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3772.078405] do_raw_spin_lock+0x177/0x190 [ 3772.078411] _raw_spin_lock+0x21/0x30 [ 3772.078416] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3772.084385] ? do_raw_spin_unlock+0x75/0x190 [ 3772.084397] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3772.084935] ? do_wait_intr+0xf0/0xf0 [ 3772.084944] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3772.085652] mdc_reint+0x2d/0x110 [mdc] [ 3772.085685] mdc_unlink+0x658/0x870 [mdc] [ 3772.085711] lmv_unlink+0x34d/0xbd0 [lmv] [ 3772.085742] ll_unlink+0x24d/0x7f0 [lustre] [ 3772.089477] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3772.089557] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3772.090964] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3772.090968] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3772.090982] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3772.090986] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3772.090991] RAX: 0000000000007f8e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3772.090994] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3772.090996] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3772.090998] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3772.091000] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3772.091003] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3772.091006] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3772.091010] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3772.091019] Call Trace: [ 3772.091025] [ 3772.091031] ? show_regs.cold.9+0x22/0x2f [ 3772.091409] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3772.091417] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3772.091422] ? watchdog+0x40/0x40 [ 3772.091426] ? hrtimer_interrupt+0x128/0x2c0 [ 3772.091430] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3772.091439] ? apic_timer_interrupt+0xf/0x20 [ 3772.091447] [ 3772.091450] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3772.091455] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3772.091463] do_raw_spin_lock+0x177/0x190 [ 3772.091468] _raw_spin_lock+0x21/0x30 [ 3772.091472] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3772.092332] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3772.094678] vfs_unlink+0x202/0x330 [ 3772.115520] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3772.115937] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3772.116204] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3772.116737] ? woken_wake_function+0x30/0x30 [ 3772.116746] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3772.117025] kthread+0x1d1/0x200 [ 3772.117032] ? set_kthread_struct+0x70/0x70 [ 3772.117036] ret_from_fork+0x1f/0x30 [ 3772.285767] do_unlinkat+0x2fa/0x460 [ 3772.287202] __x64_sys_unlink+0x37/0x50 [ 3772.288443] do_syscall_64+0xc1/0x440 [ 3772.290013] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3772.292123] RIP: 0033:0x7f83e5f5c10b [ 3772.293659] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3772.301530] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3772.304764] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3772.307353] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3772.319060] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3772.326458] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3772.334557] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3788.056536] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 3788.060190] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3788.080550] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3788.085434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3788.089699] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3788.092556] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3788.100443] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3788.103571] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3788.106876] RDX: 000000000000440b RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3788.109749] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3788.112664] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3788.116321] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3788.119234] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3788.122703] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3788.125019] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3788.127794] Call Trace: [ 3788.128865] [ 3788.129858] ? show_regs.cold.9+0x22/0x2f [ 3788.131352] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3788.133834] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3788.135564] ? watchdog+0x40/0x40 [ 3788.136990] ? hrtimer_interrupt+0x128/0x2c0 [ 3788.138946] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3788.140683] ? apic_timer_interrupt+0xf/0x20 [ 3788.142413] [ 3788.143181] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3788.145382] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3788.147837] do_raw_spin_lock+0x177/0x190 [ 3788.149317] _raw_spin_lock+0x21/0x30 [ 3788.151020] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3788.153561] obd_mod_free+0x46/0x140 [ptlrpc] [ 3788.157035] mdc_commit_open+0x91/0xb0 [mdc] [ 3788.158909] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3788.162786] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3788.166679] after_reply+0xe2c/0x1780 [ptlrpc] [ 3788.168696] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3788.171918] ? __next_timer_interrupt+0x160/0x160 [ 3788.173994] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3788.177090] ? do_wait_intr+0xf0/0xf0 [ 3788.178850] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3788.182695] kthread+0x1d1/0x200 [ 3788.184244] ? set_kthread_struct+0x70/0x70 [ 3788.185946] ret_from_fork+0x1f/0x30 [ 3792.060089] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3792.063490] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3792.081765] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3792.085763] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3792.089431] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3792.091971] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3792.094482] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3792.102978] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3792.106649] RAX: 0000000000003662 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3792.110133] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3792.113570] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3792.116870] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3792.119961] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3792.122739] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3792.126147] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3792.128763] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3792.131904] Call Trace: [ 3792.133244] [ 3792.134472] ? show_regs.cold.9+0x22/0x2f [ 3792.136506] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3792.138952] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3792.141428] ? watchdog+0x40/0x40 [ 3792.142890] ? hrtimer_interrupt+0x128/0x2c0 [ 3792.144919] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3792.147488] ? apic_timer_interrupt+0xf/0x20 [ 3792.149469] [ 3792.150400] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3792.152852] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3792.155295] do_raw_spin_lock+0x177/0x190 [ 3792.157126] _raw_spin_lock+0x21/0x30 [ 3792.158642] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3792.161059] process_one_work+0x2c8/0x700 [ 3792.162724] worker_thread+0x296/0x6e0 [ 3792.164433] ? rescuer_thread+0x570/0x570 [ 3792.166109] kthread+0x1d1/0x200 [ 3792.167639] ? set_kthread_struct+0x70/0x70 [ 3792.169510] ret_from_fork+0x1f/0x30 [ 3793.376565] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1280s! [ 3793.382110] Showing busy workqueues and worker pools: [ 3793.385398] workqueue events: flags=0x0 [ 3793.387530] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3793.390975] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 3793.396119] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3793.400073] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3793.403962] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3793.407011] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3793.411608] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3793.415683] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3793.420800] workqueue events_long: flags=0x0 [ 3793.423503] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3793.426799] pending: lu_objects_flush [obdclass] [ 3793.429891] workqueue events_unbound: flags=0x2 [ 3793.432952] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3793.436851] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3793.440511] workqueue events_power_efficient: flags=0x80 [ 3793.444117] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3793.447692] pending: neigh_periodic_work, do_cache_clean [ 3793.452117] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3793.455505] pending: neigh_periodic_work, check_lifetime [ 3793.458011] workqueue mm_percpu_wq: flags=0x8 [ 3793.460889] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3793.464744] pending: vmstat_update, lru_add_drain_per_cpu [ 3793.468121] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3793.471692] pending: vmstat_update [ 3793.473875] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3793.477086] pending: lru_add_drain_per_cpu BAR(48) [ 3793.480129] workqueue ipv6_addrconf: flags=0x40008 [ 3793.482743] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3793.485754] pending: addrconf_verify_work [ 3793.487821] workqueue knbd0-recv: flags=0x1a [ 3793.490556] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3793.493915] in-flight: 650:recv_work [ 3793.496670] workqueue ptlrpc_pinger: flags=0x2 [ 3793.500189] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3793.504816] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3793.509496] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1280s workers=4 idle: 36 466 29757 [ 3793.514016] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3783s workers=2 idle: 652 [ 3800.059070] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 3800.062273] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 3800.073036] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3800.073120] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3800.073128] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3800.073130] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3800.073145] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3800.073149] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3800.073153] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3800.073155] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3800.073158] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3800.073160] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3800.073162] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3800.073166] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3800.073168] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3800.073170] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3800.073178] Call Trace: [ 3800.073184] [ 3800.073190] ? show_regs.cold.9+0x22/0x2f [ 3800.073200] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3800.073207] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3800.073211] ? watchdog+0x40/0x40 [ 3800.073216] ? hrtimer_interrupt+0x128/0x2c0 [ 3800.073220] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3800.073226] ? apic_timer_interrupt+0xf/0x20 [ 3800.073231] [ 3800.073234] ? native_safe_halt+0x2e/0x50 [ 3800.073238] kvm_wait+0xcd/0xe0 [ 3800.073245] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3800.073253] do_raw_spin_lock+0x177/0x190 [ 3800.073257] _raw_spin_lock+0x21/0x30 [ 3800.073261] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3800.075044] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3800.084142] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3800.084225] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3800.084232] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3800.084235] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3800.084249] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3800.084254] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3800.084258] RAX: 00000000000019ef RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3800.084260] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3800.084262] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3800.084264] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3800.084267] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3800.084270] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3800.084273] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3800.084276] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3800.084284] Call Trace: [ 3800.084290] [ 3800.084296] ? show_regs.cold.9+0x22/0x2f [ 3800.084307] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3800.084314] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3800.084318] ? watchdog+0x40/0x40 [ 3800.084322] ? hrtimer_interrupt+0x128/0x2c0 [ 3800.084326] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3800.084335] ? apic_timer_interrupt+0xf/0x20 [ 3800.084340] [ 3800.084343] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3800.084348] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3800.084353] do_raw_spin_lock+0x177/0x190 [ 3800.084358] _raw_spin_lock+0x21/0x30 [ 3800.084364] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3800.084670] ? do_raw_spin_unlock+0x75/0x190 [ 3800.084874] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3800.085113] ? do_wait_intr+0xf0/0xf0 [ 3800.085121] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3800.085500] mdc_reint+0x2d/0x110 [mdc] [ 3800.085548] mdc_unlink+0x658/0x870 [mdc] [ 3800.085575] lmv_unlink+0x34d/0xbd0 [lmv] [ 3800.085611] ll_unlink+0x24d/0x7f0 [lustre] [ 3800.086825] vfs_unlink+0x202/0x330 [ 3800.086837] do_unlinkat+0x2fa/0x460 [ 3800.086842] __x64_sys_unlink+0x37/0x50 [ 3800.086845] do_syscall_64+0xc1/0x440 [ 3800.086852] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3800.087209] RIP: 0033:0x7f83e5f5c10b [ 3800.087218] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3800.087225] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3800.087232] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3800.087234] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3800.087237] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3800.087239] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3800.087241] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3800.709876] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3800.713196] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3800.719108] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3800.724266] ? woken_wake_function+0x30/0x30 [ 3800.725981] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3800.729106] kthread+0x1d1/0x200 [ 3800.730674] ? set_kthread_struct+0x70/0x70 [ 3800.734445] ret_from_fork+0x1f/0x30 [ 3820.060139] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3820.062731] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 3820.070361] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3820.070440] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3820.070447] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3820.070454] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3820.070740] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3820.070752] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3820.070756] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3820.070761] RAX: 00000000000018eb RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3820.070763] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3820.070765] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3820.070767] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3820.070769] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3820.070772] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3820.070775] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3820.070777] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3820.070786] Call Trace: [ 3820.070791] [ 3820.070797] ? show_regs.cold.9+0x22/0x2f [ 3820.070807] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3820.070814] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3820.070817] ? watchdog+0x40/0x40 [ 3820.070821] ? hrtimer_interrupt+0x128/0x2c0 [ 3820.070825] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3820.070832] ? apic_timer_interrupt+0xf/0x20 [ 3820.070837] [ 3820.070840] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3820.070844] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3820.070850] do_raw_spin_lock+0x177/0x190 [ 3820.070855] _raw_spin_lock+0x21/0x30 [ 3820.070859] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3820.072435] process_one_work+0x2c8/0x700 [ 3820.072446] worker_thread+0x296/0x6e0 [ 3820.072451] ? rescuer_thread+0x570/0x570 [ 3820.072456] kthread+0x1d1/0x200 [ 3820.072461] ? set_kthread_struct+0x70/0x70 [ 3820.072465] ret_from_fork+0x1f/0x30 [ 3820.172019] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3820.187801] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3820.191846] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3820.194678] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3820.196512] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3820.202679] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3820.204862] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3820.207074] RDX: 0000000000006a3d RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3820.209298] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3820.211373] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3820.213455] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3820.215587] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3820.218179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3820.219941] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3820.222402] Call Trace: [ 3820.223015] [ 3820.223719] ? show_regs.cold.9+0x22/0x2f [ 3820.225043] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3820.226592] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3820.228035] ? watchdog+0x40/0x40 [ 3820.229294] ? hrtimer_interrupt+0x128/0x2c0 [ 3820.230466] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3820.232080] ? apic_timer_interrupt+0xf/0x20 [ 3820.233449] [ 3820.234004] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3820.235619] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3820.237013] do_raw_spin_lock+0x177/0x190 [ 3820.238076] _raw_spin_lock+0x21/0x30 [ 3820.239007] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3820.240823] obd_mod_free+0x46/0x140 [ptlrpc] [ 3820.242416] mdc_commit_open+0x91/0xb0 [mdc] [ 3820.244954] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3820.247681] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3820.250058] after_reply+0xe2c/0x1780 [ptlrpc] [ 3820.252941] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3820.255414] ? __next_timer_interrupt+0x160/0x160 [ 3820.257386] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3820.261312] ? do_wait_intr+0xf0/0xf0 [ 3820.264718] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3820.268580] kthread+0x1d1/0x200 [ 3820.269707] ? set_kthread_struct+0x70/0x70 [ 3820.273488] ret_from_fork+0x1f/0x30 [ 3823.588712] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1310s! [ 3823.598802] Showing busy workqueues and worker pools: [ 3823.606452] workqueue events: flags=0x0 [ 3823.612225] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3823.624276] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 3823.631088] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3823.646708] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3823.658943] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3823.679984] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3823.682715] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3823.704993] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3823.719806] workqueue events_long: flags=0x0 [ 3823.724195] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3823.729416] pending: lu_objects_flush [obdclass] [ 3823.733517] workqueue events_unbound: flags=0x2 [ 3823.735890] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3823.744809] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3823.750815] workqueue events_power_efficient: flags=0x80 [ 3823.761459] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3823.768424] pending: neigh_periodic_work, do_cache_clean [ 3823.777758] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3823.785944] pending: neigh_periodic_work, check_lifetime [ 3823.796550] workqueue mm_percpu_wq: flags=0x8 [ 3823.804400] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3823.806794] pending: vmstat_update, lru_add_drain_per_cpu [ 3823.824430] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3823.847409] pending: vmstat_update [ 3823.850727] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3823.860120] pending: lru_add_drain_per_cpu BAR(48) [ 3823.865976] workqueue ipv6_addrconf: flags=0x40008 [ 3823.870836] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3823.876053] pending: addrconf_verify_work [ 3823.881154] workqueue knbd0-recv: flags=0x1a [ 3823.886342] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3823.895694] in-flight: 650:recv_work [ 3823.897197] workqueue ptlrpc_pinger: flags=0x2 [ 3823.908945] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3823.916817] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3823.926010] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1310s workers=4 idle: 36 466 29757 [ 3823.935648] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3813s workers=2 idle: 652 [ 3828.061261] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 3828.066587] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 3828.067475] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3828.067638] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3828.067644] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3828.067646] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3828.067657] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3828.067660] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3828.067663] RAX: 0000000000001c06 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3828.067665] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3828.067666] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3828.067668] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3828.067669] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3828.067673] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3828.067676] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3828.067678] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3828.067686] Call Trace: [ 3828.067691] [ 3828.067696] ? show_regs.cold.9+0x22/0x2f [ 3828.067705] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3828.067711] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3828.067716] ? watchdog+0x40/0x40 [ 3828.067720] ? hrtimer_interrupt+0x128/0x2c0 [ 3828.067724] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3828.067732] ? apic_timer_interrupt+0xf/0x20 [ 3828.067737] [ 3828.067740] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3828.067745] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3828.067751] do_raw_spin_lock+0x177/0x190 [ 3828.067756] _raw_spin_lock+0x21/0x30 [ 3828.069161] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3828.069431] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3828.071373] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3828.071634] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3828.071873] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3828.072290] ? woken_wake_function+0x30/0x30 [ 3828.072299] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3828.078872] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3828.078964] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3828.078971] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3828.078974] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3828.078988] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3828.078992] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3828.078996] RAX: 0000000000003087 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3828.078998] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3828.079000] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3828.079002] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3828.079004] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3828.079008] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3828.079010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3828.079012] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3828.079142] Call Trace: [ 3828.079153] [ 3828.079158] ? show_regs.cold.9+0x22/0x2f [ 3828.079174] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3828.079180] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3828.079185] ? watchdog+0x40/0x40 [ 3828.079188] ? hrtimer_interrupt+0x128/0x2c0 [ 3828.079192] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3828.079200] ? apic_timer_interrupt+0xf/0x20 [ 3828.079205] [ 3828.079208] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3828.079212] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3828.079217] do_raw_spin_lock+0x177/0x190 [ 3828.079221] _raw_spin_lock+0x21/0x30 [ 3828.079231] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3828.079530] ? do_raw_spin_unlock+0x75/0x190 [ 3828.079537] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3828.079744] ? do_wait_intr+0xf0/0xf0 [ 3828.079752] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3828.081194] mdc_reint+0x2d/0x110 [mdc] [ 3828.081240] mdc_unlink+0x658/0x870 [mdc] [ 3828.081265] lmv_unlink+0x34d/0xbd0 [lmv] [ 3828.081302] ll_unlink+0x24d/0x7f0 [lustre] [ 3828.081430] vfs_unlink+0x202/0x330 [ 3828.081438] do_unlinkat+0x2fa/0x460 [ 3828.081447] __x64_sys_unlink+0x37/0x50 [ 3828.081450] do_syscall_64+0xc1/0x440 [ 3828.081457] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3828.081465] RIP: 0033:0x7f83e5f5c10b [ 3828.081473] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3828.081476] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3828.081481] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3828.081483] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3828.081485] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3828.081487] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3828.081489] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3828.513313] kthread+0x1d1/0x200 [ 3828.515606] ? set_kthread_struct+0x70/0x70 [ 3828.518054] ret_from_fork+0x1f/0x30 [ 3840.996359] rcu: INFO: rcu_sched self-detected stall on CPU [ 3841.000826] rcu: 0-....: (1159144 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 3841.006419] (t=1860047 jiffies g=262441 q=57556) [ 3841.010267] rcu: rcu_sched kthread starved for 1329237 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 3841.026126] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3841.031137] rcu: RCU grace-period kthread stack dump: [ 3841.033241] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3841.039261] Call Trace: [ 3841.040180] __schedule+0x351/0xcb0 [ 3841.042102] schedule+0xc0/0x180 [ 3841.044816] schedule_timeout+0xb4/0x190 [ 3841.053329] ? do_raw_spin_unlock+0x75/0x190 [ 3841.061194] ? __next_timer_interrupt+0x160/0x160 [ 3841.062903] ? prepare_to_swait_event+0xf3/0x140 [ 3841.065073] rcu_gp_kthread+0x8dd/0x15b0 [ 3841.067389] ? force_qs_rnp+0x2c0/0x2c0 [ 3841.072820] kthread+0x1d1/0x200 [ 3841.077844] ? set_kthread_struct+0x70/0x70 [ 3841.085909] ret_from_fork+0x1f/0x30 [ 3841.091807] rcu: Stack dump where RCU GP kthread last ran: [ 3841.096919] Sending NMI from CPU 0 to CPUs 2: [ 3841.101031] NMI backtrace for cpu 2 [ 3841.101037] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3841.101039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3841.101041] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3841.101043] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3841.101044] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3841.101046] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 [ 3841.101048] RAX: 0000000000006c0c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3841.101050] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3841.101051] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3841.101053] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3841.101054] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3841.101055] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3841.101056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3841.101057] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3841.101058] Call Trace: [ 3841.101059] [ 3841.101060] ? show_regs.cold.9+0x22/0x2f [ 3841.101061] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3841.101062] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3841.101063] ? nmi_handle+0xbd/0x210 [ 3841.101064] ? default_do_nmi+0x4b/0x1b0 [ 3841.101065] ? do_nmi+0x280/0x380 [ 3841.101066] ? end_repeat_nmi+0x16/0x1f [ 3841.101067] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3841.101068] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3841.101069] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3841.101070] [ 3841.101071] do_raw_spin_lock+0x177/0x190 [ 3841.101072] _raw_spin_lock+0x21/0x30 [ 3841.101073] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3841.101075] process_one_work+0x2c8/0x700 [ 3841.101076] worker_thread+0x296/0x6e0 [ 3841.101077] ? rescuer_thread+0x570/0x570 [ 3841.101078] kthread+0x1d1/0x200 [ 3841.101079] ? set_kthread_struct+0x70/0x70 [ 3841.101079] ret_from_fork+0x1f/0x30 [ 3841.105001] NMI backtrace for cpu 0 [ 3841.273761] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3841.278401] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3841.283972] Call Trace: [ 3841.284933] [ 3841.285672] ? dump_stack+0xbb/0x10e [ 3841.289682] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3841.292528] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3841.295723] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3841.298830] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3841.304065] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3841.307362] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3841.309664] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3841.312238] ? update_process_times+0x6d/0xc0 [ 3841.316490] ? tick_sched_handle.isra.18+0x27/0x90 [ 3841.318180] ? tick_sched_timer+0x53/0xb0 [ 3841.320647] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3841.324375] ? tick_sched_handle.isra.18+0x90/0x90 [ 3841.327072] ? hrtimer_interrupt+0x128/0x2c0 [ 3841.329455] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3841.331796] ? apic_timer_interrupt+0xf/0x20 [ 3841.334264] [ 3841.335490] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3841.338469] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3841.341779] ? do_raw_spin_lock+0x177/0x190 [ 3841.345029] ? _raw_spin_lock+0x21/0x30 [ 3841.347399] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3841.351846] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3841.361830] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3841.371946] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3841.380678] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3841.390628] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3841.395104] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3841.399442] ? __next_timer_interrupt+0x160/0x160 [ 3841.403823] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3841.407542] ? do_wait_intr+0xf0/0xf0 [ 3841.411062] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3841.418339] ? kthread+0x1d1/0x200 [ 3841.420012] ? set_kthread_struct+0x70/0x70 [ 3841.422430] ? ret_from_fork+0x1f/0x30 [ 3854.307507] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1341s! [ 3854.322078] Showing busy workqueues and worker pools: [ 3854.323962] workqueue events: flags=0x0 [ 3854.329069] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3854.346099] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 3854.354058] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3854.358679] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3854.366613] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3854.371907] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3854.377114] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3854.384072] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3854.391041] workqueue events_long: flags=0x0 [ 3854.395586] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3854.402232] pending: lu_objects_flush [obdclass] [ 3854.413104] workqueue events_unbound: flags=0x2 [ 3854.417699] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3854.423315] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3854.431636] workqueue events_power_efficient: flags=0x80 [ 3854.442140] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3854.449047] pending: neigh_periodic_work, do_cache_clean [ 3854.459434] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3854.462537] pending: neigh_periodic_work, check_lifetime [ 3854.474589] workqueue mm_percpu_wq: flags=0x8 [ 3854.485590] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3854.487675] pending: vmstat_update, lru_add_drain_per_cpu [ 3854.490414] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3854.493646] pending: vmstat_update [ 3854.495501] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3854.498506] pending: lru_add_drain_per_cpu BAR(48) [ 3854.500726] workqueue ipv6_addrconf: flags=0x40008 [ 3854.502889] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3854.505369] pending: addrconf_verify_work [ 3854.507317] workqueue knbd0-recv: flags=0x1a [ 3854.509142] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3854.512312] in-flight: 650:recv_work [ 3854.515518] workqueue ptlrpc_pinger: flags=0x2 [ 3854.517833] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3854.520882] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3854.524495] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1341s workers=4 idle: 36 466 29757 [ 3854.528178] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3844s workers=2 idle: 652 [ 3856.059499] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 3856.062771] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 3856.066167] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3856.066377] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3856.066384] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3856.066388] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3856.066402] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3856.066407] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3856.066600] RAX: 000000000000775e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3856.066603] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3856.066605] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3856.066607] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3856.066609] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3856.066612] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3856.066615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3856.066618] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3856.066625] Call Trace: [ 3856.066657] [ 3856.066663] ? show_regs.cold.9+0x22/0x2f [ 3856.066672] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3856.066679] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3856.066683] ? watchdog+0x40/0x40 [ 3856.066687] ? hrtimer_interrupt+0x128/0x2c0 [ 3856.066691] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3856.066700] ? apic_timer_interrupt+0xf/0x20 [ 3856.066705] [ 3856.066708] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3856.066713] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3856.066719] do_raw_spin_lock+0x177/0x190 [ 3856.066723] _raw_spin_lock+0x21/0x30 [ 3856.066727] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3856.067040] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3856.067264] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3856.075655] Modules linked in: lustre(O) [ 3856.108206] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3856.108562] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3856.108922] ? woken_wake_function+0x30/0x30 [ 3856.108932] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3856.109150] kthread+0x1d1/0x200 [ 3856.109155] ? set_kthread_struct+0x70/0x70 [ 3856.109159] ret_from_fork+0x1f/0x30 [ 3856.308539] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3856.345637] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3856.354662] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3856.361326] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3856.366064] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3856.378530] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3856.387057] RAX: 0000000000007e91 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3856.389515] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3856.400375] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3856.405605] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3856.410105] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3856.417762] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3856.420239] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3856.427188] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3856.433534] Call Trace: [ 3856.435213] [ 3856.437205] ? show_regs.cold.9+0x22/0x2f [ 3856.438958] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3856.443377] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3856.446100] ? watchdog+0x40/0x40 [ 3856.449771] ? hrtimer_interrupt+0x128/0x2c0 [ 3856.452989] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3856.454759] ? apic_timer_interrupt+0xf/0x20 [ 3856.457615] [ 3856.459209] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3856.463238] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3856.466796] do_raw_spin_lock+0x177/0x190 [ 3856.470062] _raw_spin_lock+0x21/0x30 [ 3856.471374] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3856.477212] ? do_raw_spin_unlock+0x75/0x190 [ 3856.480602] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3856.484269] ? do_wait_intr+0xf0/0xf0 [ 3856.487184] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3856.495971] mdc_reint+0x2d/0x110 [mdc] [ 3856.504567] mdc_unlink+0x658/0x870 [mdc] [ 3856.508113] lmv_unlink+0x34d/0xbd0 [lmv] [ 3856.512138] ll_unlink+0x24d/0x7f0 [lustre] [ 3856.518088] vfs_unlink+0x202/0x330 [ 3856.520138] do_unlinkat+0x2fa/0x460 [ 3856.521616] __x64_sys_unlink+0x37/0x50 [ 3856.525602] do_syscall_64+0xc1/0x440 [ 3856.527705] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3856.533680] RIP: 0033:0x7f83e5f5c10b [ 3856.539275] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3856.551188] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3856.558161] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3856.565717] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3856.573173] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3856.581088] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3856.589783] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3868.060568] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3868.064167] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3868.104151] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3868.115122] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3868.122044] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3868.127139] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3868.130806] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3868.140518] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3868.145278] RAX: 0000000000005c8e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3868.149590] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3868.153734] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3868.156937] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3868.160958] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3868.164040] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3868.174032] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3868.179000] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3868.187515] Call Trace: [ 3868.188482] [ 3868.191232] ? show_regs.cold.9+0x22/0x2f [ 3868.192766] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3868.200855] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3868.208075] ? watchdog+0x40/0x40 [ 3868.210584] ? hrtimer_interrupt+0x128/0x2c0 [ 3868.212216] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3868.218603] ? apic_timer_interrupt+0xf/0x20 [ 3868.221020] [ 3868.222251] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3868.231285] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3868.239953] do_raw_spin_lock+0x177/0x190 [ 3868.245994] _raw_spin_lock+0x21/0x30 [ 3868.253598] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3868.261120] process_one_work+0x2c8/0x700 [ 3868.269108] worker_thread+0x296/0x6e0 [ 3868.273536] ? rescuer_thread+0x570/0x570 [ 3868.276564] kthread+0x1d1/0x200 [ 3868.278254] ? set_kthread_struct+0x70/0x70 [ 3868.280808] ret_from_fork+0x1f/0x30 [ 3880.056327] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 3880.060298] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3880.085993] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3880.092868] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3880.098463] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3880.102057] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3880.111347] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3880.114507] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3880.117248] RDX: 0000000000000b9a RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3880.120741] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3880.124294] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3880.127965] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3880.131568] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3880.135896] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3880.138965] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3880.142781] Call Trace: [ 3880.144671] [ 3880.145496] ? show_regs.cold.9+0x22/0x2f [ 3880.148075] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3880.150413] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3880.152926] ? watchdog+0x40/0x40 [ 3880.154276] ? hrtimer_interrupt+0x128/0x2c0 [ 3880.156485] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3880.158700] ? apic_timer_interrupt+0xf/0x20 [ 3880.161144] [ 3880.162064] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3880.165833] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3880.169071] do_raw_spin_lock+0x177/0x190 [ 3880.170377] _raw_spin_lock+0x21/0x30 [ 3880.172051] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3880.174743] obd_mod_free+0x46/0x140 [ptlrpc] [ 3880.177260] mdc_commit_open+0x91/0xb0 [mdc] [ 3880.179210] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3880.182246] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3880.185437] after_reply+0xe2c/0x1780 [ptlrpc] [ 3880.188727] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3880.192378] ? __next_timer_interrupt+0x160/0x160 [ 3880.194668] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3880.197332] ? do_wait_intr+0xf0/0xf0 [ 3880.199645] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3880.202259] kthread+0x1d1/0x200 [ 3880.203669] ? set_kthread_struct+0x70/0x70 [ 3880.205126] ret_from_fork+0x1f/0x30 [ 3884.061185] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 3884.062021] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 3884.071891] Modules linked in: [ 3884.075560] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3884.075644] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3884.075654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3884.075657] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3884.075671] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3884.075676] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3884.075681] RAX: 0000000000000d7e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3884.075684] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3884.075686] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3884.075688] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3884.075691] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3884.075694] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3884.075698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3884.075700] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3884.075710] Call Trace: [ 3884.075715] [ 3884.075720] ? show_regs.cold.9+0x22/0x2f [ 3884.075730] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3884.075736] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3884.075741] ? watchdog+0x40/0x40 [ 3884.075745] ? hrtimer_interrupt+0x128/0x2c0 [ 3884.075749] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3884.075758] ? apic_timer_interrupt+0xf/0x20 [ 3884.075763] [ 3884.075766] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3884.075771] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3884.075776] do_raw_spin_lock+0x177/0x190 [ 3884.075781] _raw_spin_lock+0x21/0x30 [ 3884.075786] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3884.077739] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) [ 3884.098834] ? do_raw_spin_unlock+0x75/0x190 [ 3884.104113] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3884.104188] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3884.104195] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3884.104198] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3884.104212] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3884.104215] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3884.104220] RAX: 0000000000007e17 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3884.104222] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3884.104224] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3884.104226] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3884.104229] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3884.104232] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3884.104234] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3884.104236] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3884.104244] Call Trace: [ 3884.104251] [ 3884.104257] ? show_regs.cold.9+0x22/0x2f [ 3884.104290] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3884.104298] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3884.104306] ? watchdog+0x40/0x40 [ 3884.104311] ? hrtimer_interrupt+0x128/0x2c0 [ 3884.104315] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3884.104323] ? apic_timer_interrupt+0xf/0x20 [ 3884.104327] [ 3884.104331] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3884.104335] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3884.104342] do_raw_spin_lock+0x177/0x190 [ 3884.104347] _raw_spin_lock+0x21/0x30 [ 3884.104355] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3884.108676] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3884.111772] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3884.120434] ? do_wait_intr+0xf0/0xf0 [ 3884.125061] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3884.127973] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3884.131344] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3884.134744] mdc_reint+0x2d/0x110 [mdc] [ 3884.138126] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3884.141594] mdc_unlink+0x658/0x870 [mdc] [ 3884.141656] lmv_unlink+0x34d/0xbd0 [lmv] [ 3884.141695] ll_unlink+0x24d/0x7f0 [lustre] [ 3884.141823] vfs_unlink+0x202/0x330 [ 3884.141831] do_unlinkat+0x2fa/0x460 [ 3884.141837] __x64_sys_unlink+0x37/0x50 [ 3884.141840] do_syscall_64+0xc1/0x440 [ 3884.141846] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3884.141856] RIP: 0033:0x7f83e5f5c10b [ 3884.141863] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3884.141867] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3884.141873] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3884.141875] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3884.141877] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3884.141879] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3884.141881] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3884.332429] ? woken_wake_function+0x30/0x30 [ 3884.336928] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3884.342427] kthread+0x1d1/0x200 [ 3884.345179] ? set_kthread_struct+0x70/0x70 [ 3884.350686] ret_from_fork+0x1f/0x30 [ 3885.024070] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1371s! [ 3885.030355] Showing busy workqueues and worker pools: [ 3885.034591] workqueue events: flags=0x0 [ 3885.036211] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3885.039385] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 3885.043288] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3885.046701] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3885.050095] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3885.052898] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3885.057585] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3885.060533] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3885.065398] workqueue events_long: flags=0x0 [ 3885.067135] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3885.069750] pending: lu_objects_flush [obdclass] [ 3885.072206] workqueue events_unbound: flags=0x2 [ 3885.074366] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3885.076776] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3885.080167] workqueue events_power_efficient: flags=0x80 [ 3885.082273] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3885.084189] pending: neigh_periodic_work, do_cache_clean [ 3885.086400] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3885.089658] pending: neigh_periodic_work, check_lifetime [ 3885.092579] workqueue mm_percpu_wq: flags=0x8 [ 3885.094406] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3885.097843] pending: vmstat_update, lru_add_drain_per_cpu [ 3885.100767] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3885.104075] pending: vmstat_update [ 3885.106320] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3885.109007] pending: lru_add_drain_per_cpu BAR(48) [ 3885.110893] workqueue ipv6_addrconf: flags=0x40008 [ 3885.112566] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3885.114937] pending: addrconf_verify_work [ 3885.116392] workqueue knbd0-recv: flags=0x1a [ 3885.118051] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3885.121258] in-flight: 650:recv_work [ 3885.123177] workqueue ptlrpc_pinger: flags=0x2 [ 3885.125297] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3885.127330] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3885.130117] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1371s workers=4 idle: 36 466 29757 [ 3885.134110] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3875s workers=2 idle: 652 [ 3896.060057] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3896.064276] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3896.091748] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3896.098495] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3896.103593] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3896.107534] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3896.110725] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3896.120228] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3896.124018] RAX: 0000000000004d82 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3896.128085] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3896.132024] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3896.136112] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3896.140316] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3896.144337] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3896.148544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3896.151720] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3896.156262] Call Trace: [ 3896.158049] [ 3896.158728] ? show_regs.cold.9+0x22/0x2f [ 3896.161084] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3896.164038] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3896.166535] ? watchdog+0x40/0x40 [ 3896.167882] ? hrtimer_interrupt+0x128/0x2c0 [ 3896.169997] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3896.172841] ? apic_timer_interrupt+0xf/0x20 [ 3896.175289] [ 3896.176584] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3896.179221] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3896.182391] do_raw_spin_lock+0x177/0x190 [ 3896.184731] _raw_spin_lock+0x21/0x30 [ 3896.186637] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3896.190070] process_one_work+0x2c8/0x700 [ 3896.192431] worker_thread+0x296/0x6e0 [ 3896.194559] ? rescuer_thread+0x570/0x570 [ 3896.197208] kthread+0x1d1/0x200 [ 3896.199222] ? set_kthread_struct+0x70/0x70 [ 3896.201475] ret_from_fork+0x1f/0x30 [ 3912.059140] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 3912.059559] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 3912.062017] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 3912.062030] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3912.062098] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3912.062104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3912.062107] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3912.062121] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3912.062125] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3912.062129] RAX: 0000000000006f25 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3912.062131] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3912.062134] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3912.062136] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3912.062139] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3912.062143] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3912.062146] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3912.062148] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3912.062157] Call Trace: [ 3912.062163] [ 3912.062169] ? show_regs.cold.9+0x22/0x2f [ 3912.062178] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3912.062185] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3912.062189] ? watchdog+0x40/0x40 [ 3912.062193] ? hrtimer_interrupt+0x128/0x2c0 [ 3912.062197] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3912.062206] ? apic_timer_interrupt+0xf/0x20 [ 3912.062210] [ 3912.062213] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3912.062218] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3912.062237] do_raw_spin_lock+0x177/0x190 [ 3912.062242] _raw_spin_lock+0x21/0x30 [ 3912.062246] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3912.062758] ? do_raw_spin_unlock+0x75/0x190 [ 3912.062764] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3912.063194] ? do_wait_intr+0xf0/0xf0 [ 3912.063202] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3912.063536] mdc_reint+0x2d/0x110 [mdc] [ 3912.063581] mdc_unlink+0x658/0x870 [mdc] [ 3912.063609] lmv_unlink+0x34d/0xbd0 [lmv] [ 3912.063645] ll_unlink+0x24d/0x7f0 [lustre] [ 3912.063780] vfs_unlink+0x202/0x330 [ 3912.063983] do_unlinkat+0x2fa/0x460 [ 3912.063988] __x64_sys_unlink+0x37/0x50 [ 3912.063992] do_syscall_64+0xc1/0x440 [ 3912.063999] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3912.064018] RIP: 0033:0x7f83e5f5c10b [ 3912.064026] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3912.064030] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3912.064035] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3912.064038] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3912.064040] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3912.064042] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3912.064044] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3912.064970] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3912.065115] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3912.065123] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3912.065125] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3912.065137] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3912.065141] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3912.065146] RAX: 000000000000328a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3912.065149] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3912.065151] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3912.065153] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3912.065156] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3912.065159] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3912.065162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3912.065164] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3912.065172] Call Trace: [ 3912.065176] [ 3912.065180] ? show_regs.cold.9+0x22/0x2f [ 3912.065188] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3912.065194] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3912.065198] ? watchdog+0x40/0x40 [ 3912.065274] ? hrtimer_interrupt+0x128/0x2c0 [ 3912.065281] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3912.065288] ? apic_timer_interrupt+0xf/0x20 [ 3912.065294] [ 3912.065298] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3912.065304] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3912.065312] do_raw_spin_lock+0x177/0x190 [ 3912.065319] _raw_spin_lock+0x21/0x30 [ 3912.065325] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3912.065608] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3912.067804] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3912.082621] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3912.083670] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3912.083677] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3912.083680] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3912.083693] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3912.083697] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3912.083701] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3912.083703] RDX: 000000000000554e RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3912.083705] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3912.083708] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3912.083710] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3912.083712] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3912.083715] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3912.083717] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3912.083725] Call Trace: [ 3912.083731] [ 3912.083736] ? show_regs.cold.9+0x22/0x2f [ 3912.083746] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3912.083753] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3912.083757] ? watchdog+0x40/0x40 [ 3912.083761] ? hrtimer_interrupt+0x128/0x2c0 [ 3912.083765] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3912.083774] ? apic_timer_interrupt+0xf/0x20 [ 3912.083778] [ 3912.083781] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3912.083786] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3912.083791] do_raw_spin_lock+0x177/0x190 [ 3912.083796] _raw_spin_lock+0x21/0x30 [ 3912.083801] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3912.084251] obd_mod_free+0x46/0x140 [ptlrpc] [ 3912.084469] mdc_commit_open+0x91/0xb0 [mdc] [ 3912.084507] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3912.084720] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3912.084928] after_reply+0xe2c/0x1780 [ptlrpc] [ 3912.089826] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3912.090623] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3912.090842] ? woken_wake_function+0x30/0x30 [ 3912.090852] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3912.091401] kthread+0x1d1/0x200 [ 3912.091408] ? set_kthread_struct+0x70/0x70 [ 3912.091412] ret_from_fork+0x1f/0x30 [ 3913.201495] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3913.208630] ? __next_timer_interrupt+0x160/0x160 [ 3913.213832] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3913.219959] ? do_wait_intr+0xf0/0xf0 [ 3913.222442] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3913.232019] kthread+0x1d1/0x200 [ 3913.236277] ? set_kthread_struct+0x70/0x70 [ 3913.244389] ret_from_fork+0x1f/0x30 [ 3915.234521] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1402s! [ 3915.238668] Showing busy workqueues and worker pools: [ 3915.240664] workqueue events: flags=0x0 [ 3915.243113] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3915.246679] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 3915.250730] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3915.256425] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3915.260490] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3915.263478] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3915.267046] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3915.270703] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3915.275159] workqueue events_long: flags=0x0 [ 3915.276484] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3915.281331] pending: lu_objects_flush [obdclass] [ 3915.283714] workqueue events_unbound: flags=0x2 [ 3915.285588] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3915.290731] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3915.295433] workqueue events_power_efficient: flags=0x80 [ 3915.301330] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3915.304768] pending: neigh_periodic_work, do_cache_clean [ 3915.308687] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3915.311597] pending: neigh_periodic_work, check_lifetime [ 3915.318509] workqueue mm_percpu_wq: flags=0x8 [ 3915.320920] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3915.323772] pending: vmstat_update, lru_add_drain_per_cpu [ 3915.329612] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3915.334251] pending: vmstat_update [ 3915.337442] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3915.341803] pending: lru_add_drain_per_cpu BAR(48) [ 3915.344413] workqueue ipv6_addrconf: flags=0x40008 [ 3915.350094] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3915.354816] pending: addrconf_verify_work [ 3915.359492] workqueue knbd0-recv: flags=0x1a [ 3915.363114] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3915.368057] in-flight: 650:recv_work [ 3915.373543] workqueue ptlrpc_pinger: flags=0x2 [ 3915.375847] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3915.378550] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3915.383183] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1402s workers=4 idle: 36 466 29757 [ 3915.387416] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3905s workers=2 idle: 652 [ 3924.060037] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3924.065111] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3924.086779] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3924.092338] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3924.096529] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3924.099306] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3924.102159] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3924.110568] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3924.114034] RAX: 0000000000006b66 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3924.117077] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3924.120096] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3924.123426] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3924.125952] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3924.128858] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3924.132935] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3924.135390] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3924.137509] Call Trace: [ 3924.139026] [ 3924.140009] ? show_regs.cold.9+0x22/0x2f [ 3924.141763] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3924.143786] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3924.147058] ? watchdog+0x40/0x40 [ 3924.149024] ? hrtimer_interrupt+0x128/0x2c0 [ 3924.153920] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3924.157107] ? apic_timer_interrupt+0xf/0x20 [ 3924.160386] [ 3924.161918] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3924.164450] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3924.167452] do_raw_spin_lock+0x177/0x190 [ 3924.169330] _raw_spin_lock+0x21/0x30 [ 3924.171072] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3924.173796] process_one_work+0x2c8/0x700 [ 3924.175730] worker_thread+0x296/0x6e0 [ 3924.177716] ? rescuer_thread+0x570/0x570 [ 3924.179585] kthread+0x1d1/0x200 [ 3924.181252] ? set_kthread_struct+0x70/0x70 [ 3924.183561] ret_from_fork+0x1f/0x30 [ 3940.060825] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 3940.062278] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 3940.064295] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 3940.064309] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3940.064383] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3940.064389] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3940.064392] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3940.064405] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3940.064409] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3940.064413] RAX: 0000000000003592 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3940.064415] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3940.064417] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3940.064419] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3940.064422] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3940.064425] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3940.064428] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3940.064430] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3940.064438] Call Trace: [ 3940.064443] [ 3940.064449] ? show_regs.cold.9+0x22/0x2f [ 3940.064458] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3940.064464] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3940.064469] ? watchdog+0x40/0x40 [ 3940.064473] ? hrtimer_interrupt+0x128/0x2c0 [ 3940.064477] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3940.064486] ? apic_timer_interrupt+0xf/0x20 [ 3940.064491] [ 3940.064494] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3940.064580] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3940.064585] do_raw_spin_lock+0x177/0x190 [ 3940.064590] _raw_spin_lock+0x21/0x30 [ 3940.064596] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3940.064968] ? do_raw_spin_unlock+0x75/0x190 [ 3940.064974] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3940.065464] ? do_wait_intr+0xf0/0xf0 [ 3940.065472] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3940.065665] mdc_reint+0x2d/0x110 [mdc] [ 3940.065779] mdc_unlink+0x658/0x870 [mdc] [ 3940.065806] lmv_unlink+0x34d/0xbd0 [lmv] [ 3940.065838] ll_unlink+0x24d/0x7f0 [lustre] [ 3940.065946] vfs_unlink+0x202/0x330 [ 3940.065954] do_unlinkat+0x2fa/0x460 [ 3940.065958] __x64_sys_unlink+0x37/0x50 [ 3940.065961] do_syscall_64+0xc1/0x440 [ 3940.065967] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3940.065974] RIP: 0033:0x7f83e5f5c10b [ 3940.065981] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3940.065984] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3940.065989] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3940.065991] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3940.065992] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3940.065994] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3940.065996] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3940.068573] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3940.068662] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3940.068670] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3940.068673] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3940.068819] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3940.068823] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3940.068827] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3940.068829] RDX: 0000000000004c09 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3940.068832] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3940.068833] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3940.068835] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3940.068838] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3940.068840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3940.068842] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3940.068860] Call Trace: [ 3940.068866] [ 3940.068871] ? show_regs.cold.9+0x22/0x2f [ 3940.068880] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3940.068886] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3940.068890] ? watchdog+0x40/0x40 [ 3940.068894] ? hrtimer_interrupt+0x128/0x2c0 [ 3940.068897] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3940.068906] ? apic_timer_interrupt+0xf/0x20 [ 3940.068911] [ 3940.068914] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3940.068918] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3940.068922] do_raw_spin_lock+0x177/0x190 [ 3940.068927] _raw_spin_lock+0x21/0x30 [ 3940.068934] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3940.069215] obd_mod_free+0x46/0x140 [ptlrpc] [ 3940.069913] mdc_commit_open+0x91/0xb0 [mdc] [ 3940.069960] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3940.070178] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3940.071581] after_reply+0xe2c/0x1780 [ptlrpc] [ 3940.073274] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3940.074918] ? __next_timer_interrupt+0x160/0x160 [ 3940.074929] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3940.075201] ? do_wait_intr+0xf0/0xf0 [ 3940.075211] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3940.075415] kthread+0x1d1/0x200 [ 3940.075421] ? set_kthread_struct+0x70/0x70 [ 3940.075425] ret_from_fork+0x1f/0x30 [ 3940.640217] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3940.686139] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3940.700890] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3940.709579] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3940.716029] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3940.735800] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3940.738538] RAX: 0000000000005c0f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3940.743324] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3940.754374] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3940.761779] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3940.769359] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3940.771676] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3940.781069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3940.783396] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3940.788741] Call Trace: [ 3940.789580] [ 3940.790105] ? show_regs.cold.9+0x22/0x2f [ 3940.800879] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3940.803537] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3940.804707] ? watchdog+0x40/0x40 [ 3940.806333] ? hrtimer_interrupt+0x128/0x2c0 [ 3940.809501] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3940.816623] ? apic_timer_interrupt+0xf/0x20 [ 3940.820445] [ 3940.821551] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3940.831846] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3940.834257] do_raw_spin_lock+0x177/0x190 [ 3940.836897] _raw_spin_lock+0x21/0x30 [ 3940.844475] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3940.849737] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3940.855238] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3940.862990] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3940.871313] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3940.873559] ? woken_wake_function+0x30/0x30 [ 3940.877486] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3940.881086] kthread+0x1d1/0x200 [ 3940.882293] ? set_kthread_struct+0x70/0x70 [ 3940.887247] ret_from_fork+0x1f/0x30 [ 3945.445862] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1432s! [ 3945.450717] Showing busy workqueues and worker pools: [ 3945.457380] workqueue events: flags=0x0 [ 3945.460542] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3945.468812] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 3945.473651] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3945.482017] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3945.489950] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3945.497076] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3945.511271] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3945.517383] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3945.530390] workqueue events_long: flags=0x0 [ 3945.536671] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3945.546103] pending: lu_objects_flush [obdclass] [ 3945.555354] workqueue events_unbound: flags=0x2 [ 3945.565708] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3945.573038] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3945.575335] workqueue events_power_efficient: flags=0x80 [ 3945.583407] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3945.587150] pending: neigh_periodic_work, do_cache_clean [ 3945.590075] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3945.601300] pending: neigh_periodic_work, check_lifetime [ 3945.609423] workqueue mm_percpu_wq: flags=0x8 [ 3945.613116] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3945.619742] pending: vmstat_update, lru_add_drain_per_cpu [ 3945.621707] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3945.628839] pending: vmstat_update [ 3945.631503] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3945.633794] pending: lru_add_drain_per_cpu BAR(48) [ 3945.636391] workqueue ipv6_addrconf: flags=0x40008 [ 3945.639072] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3945.643859] pending: addrconf_verify_work [ 3945.651267] workqueue knbd0-recv: flags=0x1a [ 3945.657059] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3945.663081] in-flight: 650:recv_work [ 3945.665702] workqueue ptlrpc_pinger: flags=0x2 [ 3945.669786] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3945.676702] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3945.680825] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1432s workers=4 idle: 36 466 29757 [ 3945.693157] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3935s workers=2 idle: 652 [ 3952.060361] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3952.066669] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3952.101086] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3952.113118] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3952.123416] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3952.127595] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3952.132308] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3952.147398] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3952.153291] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3952.158424] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3952.164927] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3952.172287] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3952.176946] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3952.180349] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3952.185108] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3952.187984] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3952.192189] Call Trace: [ 3952.193669] [ 3952.194603] ? show_regs.cold.9+0x22/0x2f [ 3952.196264] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3952.198867] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3952.201076] ? watchdog+0x40/0x40 [ 3952.202658] ? hrtimer_interrupt+0x128/0x2c0 [ 3952.204828] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3952.206963] ? apic_timer_interrupt+0xf/0x20 [ 3952.208989] [ 3952.209821] ? native_safe_halt+0x2e/0x50 [ 3952.211911] kvm_wait+0xcd/0xe0 [ 3952.213336] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3952.215962] do_raw_spin_lock+0x177/0x190 [ 3952.217444] _raw_spin_lock+0x21/0x30 [ 3952.219097] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3952.221655] process_one_work+0x2c8/0x700 [ 3952.223209] worker_thread+0x296/0x6e0 [ 3952.226377] ? rescuer_thread+0x570/0x570 [ 3952.228123] kthread+0x1d1/0x200 [ 3952.229813] ? set_kthread_struct+0x70/0x70 [ 3952.231181] ret_from_fork+0x1f/0x30 [ 3968.059050] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 3968.061134] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3968.062023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 3968.078453] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3968.078474] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3968.078477] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3968.078490] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3968.078494] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3968.078500] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3968.078502] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3968.078503] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3968.078505] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3968.078507] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3968.078510] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3968.078512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3968.078514] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3968.078521] Call Trace: [ 3968.078526] [ 3968.078531] ? show_regs.cold.9+0x22/0x2f [ 3968.078539] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3968.078544] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3968.078548] ? watchdog+0x40/0x40 [ 3968.078552] ? hrtimer_interrupt+0x128/0x2c0 [ 3968.078556] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3968.078563] ? apic_timer_interrupt+0xf/0x20 [ 3968.078568] [ 3968.078571] ? native_safe_halt+0x2e/0x50 [ 3968.078575] kvm_wait+0xcd/0xe0 [ 3968.078583] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3968.078592] do_raw_spin_lock+0x177/0x190 [ 3968.078596] _raw_spin_lock+0x21/0x30 [ 3968.078600] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3968.083293] Modules linked in: lustre(O) [ 3968.086865] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3968.095017] osp(O) [ 3968.096529] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3968.117548] ofd(O) [ 3968.119981] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3968.123597] lod(O) [ 3968.125595] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3968.128568] mdt(O) mdd(O) [ 3968.131553] ? woken_wake_function+0x30/0x30 [ 3968.135310] mgs(O) [ 3968.138000] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3968.140308] lquota(O) [ 3968.144274] kthread+0x1d1/0x200 [ 3968.145858] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) [ 3968.146407] ? set_kthread_struct+0x70/0x70 [ 3968.146417] ret_from_fork+0x1f/0x30 [ 3968.200073] fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3968.218356] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3968.224080] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3968.227786] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3968.230272] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3968.240889] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3968.243412] RAX: 0000000000006c12 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3968.245868] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3968.248358] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3968.252085] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3968.254449] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3968.257083] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3968.259554] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3968.261478] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3968.264797] Call Trace: [ 3968.265634] [ 3968.266476] ? show_regs.cold.9+0x22/0x2f [ 3968.267936] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3968.269641] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3968.271355] ? watchdog+0x40/0x40 [ 3968.272540] ? hrtimer_interrupt+0x128/0x2c0 [ 3968.274048] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3968.275507] ? apic_timer_interrupt+0xf/0x20 [ 3968.277042] [ 3968.277645] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3968.279750] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3968.281485] do_raw_spin_lock+0x177/0x190 [ 3968.282638] _raw_spin_lock+0x21/0x30 [ 3968.283683] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3968.285486] ? do_raw_spin_unlock+0x75/0x190 [ 3968.286912] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3968.288773] ? do_wait_intr+0xf0/0xf0 [ 3968.292206] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3968.298515] mdc_reint+0x2d/0x110 [mdc] [ 3968.302913] mdc_unlink+0x658/0x870 [mdc] [ 3968.306557] lmv_unlink+0x34d/0xbd0 [lmv] [ 3968.311509] ll_unlink+0x24d/0x7f0 [lustre] [ 3968.317327] vfs_unlink+0x202/0x330 [ 3968.320312] do_unlinkat+0x2fa/0x460 [ 3968.321339] __x64_sys_unlink+0x37/0x50 [ 3968.322307] do_syscall_64+0xc1/0x440 [ 3968.324143] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3968.326402] RIP: 0033:0x7f83e5f5c10b [ 3968.329502] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3968.338532] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3968.341377] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3968.347411] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3968.349625] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3968.352432] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3968.355970] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 3972.059525] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 3972.061919] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3972.086066] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3972.091271] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3972.095586] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3972.101578] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3972.114905] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3972.119982] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 3972.123390] RDX: 0000000000004b0f RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 3972.127229] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 3972.130710] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 3972.136448] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3972.140543] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 3972.144166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3972.146568] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 3972.154891] Call Trace: [ 3972.155597] [ 3972.156249] ? show_regs.cold.9+0x22/0x2f [ 3972.160928] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3972.162520] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3972.166033] ? watchdog+0x40/0x40 [ 3972.167106] ? hrtimer_interrupt+0x128/0x2c0 [ 3972.168992] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3972.173170] ? apic_timer_interrupt+0xf/0x20 [ 3972.175014] [ 3972.175820] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3972.178233] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3972.180722] do_raw_spin_lock+0x177/0x190 [ 3972.182097] _raw_spin_lock+0x21/0x30 [ 3972.183839] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3972.186261] obd_mod_free+0x46/0x140 [ptlrpc] [ 3972.188472] mdc_commit_open+0x91/0xb0 [mdc] [ 3972.190180] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3972.192771] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3972.195316] after_reply+0xe2c/0x1780 [ptlrpc] [ 3972.197568] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3972.199865] ? __next_timer_interrupt+0x160/0x160 [ 3972.201406] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3972.203063] ? do_wait_intr+0xf0/0xf0 [ 3972.204556] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3972.206388] kthread+0x1d1/0x200 [ 3972.207399] ? set_kthread_struct+0x70/0x70 [ 3972.208855] ret_from_fork+0x1f/0x30 [ 3976.161445] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1462s! [ 3976.165019] Showing busy workqueues and worker pools: [ 3976.167508] workqueue events: flags=0x0 [ 3976.169200] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3976.172107] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 3976.175778] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3976.178701] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 3976.181797] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3976.185169] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 3976.189274] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3976.193127] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 3976.196625] workqueue events_long: flags=0x0 [ 3976.198487] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3976.201963] pending: lu_objects_flush [obdclass] [ 3976.204884] workqueue events_unbound: flags=0x2 [ 3976.206961] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3976.209991] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3976.213090] workqueue events_power_efficient: flags=0x80 [ 3976.215780] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3976.219251] pending: neigh_periodic_work, do_cache_clean [ 3976.222117] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3976.226515] pending: neigh_periodic_work, check_lifetime [ 3976.229015] workqueue mm_percpu_wq: flags=0x8 [ 3976.231030] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3976.234555] pending: vmstat_update, lru_add_drain_per_cpu [ 3976.237731] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3976.240984] pending: vmstat_update [ 3976.243214] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 3976.247356] pending: lru_add_drain_per_cpu BAR(48) [ 3976.250233] workqueue ipv6_addrconf: flags=0x40008 [ 3976.252743] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3976.255980] pending: addrconf_verify_work [ 3976.257851] workqueue knbd0-recv: flags=0x1a [ 3976.260298] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3976.263881] in-flight: 650:recv_work [ 3976.265844] workqueue ptlrpc_pinger: flags=0x2 [ 3976.268222] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3976.271658] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 3976.274657] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1463s workers=4 idle: 36 466 29757 [ 3976.278390] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3966s workers=2 idle: 652 [ 3980.060025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 3980.062697] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3980.092506] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3980.100368] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3980.107925] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3980.114246] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3980.117717] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3980.133133] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3980.138991] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3980.143919] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 3980.147578] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 3980.154264] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 3980.160666] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 3980.165114] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 3980.169952] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3980.173515] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 3980.178265] Call Trace: [ 3980.180770] [ 3980.181316] ? show_regs.cold.9+0x22/0x2f [ 3980.185763] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3980.187505] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3980.190715] ? watchdog+0x40/0x40 [ 3980.192438] ? hrtimer_interrupt+0x128/0x2c0 [ 3980.194815] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3980.196910] ? apic_timer_interrupt+0xf/0x20 [ 3980.199107] [ 3980.200091] ? native_safe_halt+0x2e/0x50 [ 3980.202162] kvm_wait+0xcd/0xe0 [ 3980.203855] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3980.206526] do_raw_spin_lock+0x177/0x190 [ 3980.208859] _raw_spin_lock+0x21/0x30 [ 3980.210484] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3980.213321] process_one_work+0x2c8/0x700 [ 3980.215270] worker_thread+0x296/0x6e0 [ 3980.217328] ? rescuer_thread+0x570/0x570 [ 3980.219918] kthread+0x1d1/0x200 [ 3980.221728] ? set_kthread_struct+0x70/0x70 [ 3980.223840] ret_from_fork+0x1f/0x30 [ 3996.059153] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 3996.061966] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3996.067137] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 3996.083049] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3996.083270] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3996.083274] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3996.083289] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3996.083315] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3996.083321] RAX: 0000000000005888 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3996.083323] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 3996.083325] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3996.083327] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 3996.083330] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 3996.083333] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 3996.083340] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3996.083343] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 3996.083350] Call Trace: [ 3996.083356] [ 3996.083362] ? show_regs.cold.9+0x22/0x2f [ 3996.083372] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3996.083496] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3996.083501] ? watchdog+0x40/0x40 [ 3996.083504] ? hrtimer_interrupt+0x128/0x2c0 [ 3996.083508] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3996.083516] ? apic_timer_interrupt+0xf/0x20 [ 3996.083521] [ 3996.083524] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3996.083528] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3996.083536] do_raw_spin_lock+0x177/0x190 [ 3996.083541] _raw_spin_lock+0x21/0x30 [ 3996.083546] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3996.083814] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3996.084004] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3996.092516] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3996.092594] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3996.092601] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3996.092604] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3996.092618] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3996.092622] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3996.092626] RAX: 0000000000002f84 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 3996.092628] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 3996.092629] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3996.092631] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 3996.092633] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 3996.092636] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 3996.092639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3996.092640] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 3996.092649] Call Trace: [ 3996.092655] [ 3996.092661] ? show_regs.cold.9+0x22/0x2f [ 3996.092670] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3996.092676] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3996.092680] ? watchdog+0x40/0x40 [ 3996.092684] ? hrtimer_interrupt+0x128/0x2c0 [ 3996.092688] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3996.092697] ? apic_timer_interrupt+0xf/0x20 [ 3996.092701] [ 3996.092705] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3996.092710] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3996.092715] do_raw_spin_lock+0x177/0x190 [ 3996.092719] _raw_spin_lock+0x21/0x30 [ 3996.092725] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3996.092993] ? do_raw_spin_unlock+0x75/0x190 [ 3996.093000] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3996.093368] ? do_wait_intr+0xf0/0xf0 [ 3996.093376] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3996.093749] mdc_reint+0x2d/0x110 [mdc] [ 3996.099301] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3996.099757] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3996.100131] ? woken_wake_function+0x30/0x30 [ 3996.100141] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3996.100435] kthread+0x1d1/0x200 [ 3996.100442] ? set_kthread_struct+0x70/0x70 [ 3996.100446] ret_from_fork+0x1f/0x30 [ 3996.543207] mdc_unlink+0x658/0x870 [mdc] [ 3996.545851] lmv_unlink+0x34d/0xbd0 [lmv] [ 3996.548034] ll_unlink+0x24d/0x7f0 [lustre] [ 3996.550695] vfs_unlink+0x202/0x330 [ 3996.552681] do_unlinkat+0x2fa/0x460 [ 3996.554842] __x64_sys_unlink+0x37/0x50 [ 3996.557200] do_syscall_64+0xc1/0x440 [ 3996.559492] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3996.562735] RIP: 0033:0x7f83e5f5c10b [ 3996.564674] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 3996.575241] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3996.579092] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 3996.586175] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 3996.591953] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 3996.599484] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 3996.608039] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4004.060730] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 4004.066302] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4004.100674] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4004.108462] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4004.121484] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4004.129647] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4004.145058] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4004.150239] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4004.154801] RDX: 0000000000004baa RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4004.159533] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4004.163272] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4004.167596] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4004.170955] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4004.175406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4004.178416] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4004.182104] Call Trace: [ 4004.183388] [ 4004.184308] ? show_regs.cold.9+0x22/0x2f [ 4004.186413] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4004.189118] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4004.191184] ? watchdog+0x40/0x40 [ 4004.193229] ? hrtimer_interrupt+0x128/0x2c0 [ 4004.195155] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4004.197856] ? apic_timer_interrupt+0xf/0x20 [ 4004.200587] [ 4004.201750] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4004.204814] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4004.207302] do_raw_spin_lock+0x177/0x190 [ 4004.209306] _raw_spin_lock+0x21/0x30 [ 4004.211178] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4004.214247] obd_mod_free+0x46/0x140 [ptlrpc] [ 4004.216541] mdc_commit_open+0x91/0xb0 [mdc] [ 4004.219085] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4004.221769] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4004.225209] after_reply+0xe2c/0x1780 [ptlrpc] [ 4004.227881] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4004.230794] ? __next_timer_interrupt+0x160/0x160 [ 4004.233174] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4004.235696] ? do_wait_intr+0xf0/0xf0 [ 4004.237860] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4004.240999] kthread+0x1d1/0x200 [ 4004.242896] ? set_kthread_struct+0x70/0x70 [ 4004.244683] ret_from_fork+0x1f/0x30 [ 4006.371107] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1493s! [ 4006.376690] Showing busy workqueues and worker pools: [ 4006.381624] workqueue events: flags=0x0 [ 4006.384643] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4006.390547] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4006.398621] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4006.404181] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4006.407634] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4006.410652] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4006.419451] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4006.423246] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4006.427494] workqueue events_long: flags=0x0 [ 4006.429416] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4006.432641] pending: lu_objects_flush [obdclass] [ 4006.438923] workqueue events_unbound: flags=0x2 [ 4006.440679] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4006.444607] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4006.447664] workqueue events_power_efficient: flags=0x80 [ 4006.453165] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4006.457098] pending: neigh_periodic_work, do_cache_clean [ 4006.460101] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4006.462663] pending: neigh_periodic_work, check_lifetime [ 4006.465777] workqueue mm_percpu_wq: flags=0x8 [ 4006.467548] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4006.470247] pending: vmstat_update, lru_add_drain_per_cpu [ 4006.473016] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4006.475887] pending: vmstat_update [ 4006.477758] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4006.480659] pending: lru_add_drain_per_cpu BAR(48) [ 4006.483178] workqueue ipv6_addrconf: flags=0x40008 [ 4006.485552] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4006.489553] pending: addrconf_verify_work [ 4006.493101] workqueue knbd0-recv: flags=0x1a [ 4006.495261] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4006.498235] in-flight: 650:recv_work [ 4006.501154] workqueue ptlrpc_pinger: flags=0x2 [ 4006.502657] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4006.508470] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4006.512352] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1493s workers=4 idle: 36 466 29757 [ 4006.516944] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3996s workers=2 idle: 652 [ 4008.065193] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 4008.070416] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4008.111785] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4008.128647] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4008.135760] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4008.142178] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4008.145763] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4008.163030] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4008.172718] RAX: 000000000000548a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4008.181971] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4008.188089] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4008.193772] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4008.205546] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4008.210717] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4008.215409] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4008.223956] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4008.231614] Call Trace: [ 4008.237217] [ 4008.237968] ? show_regs.cold.9+0x22/0x2f [ 4008.243268] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4008.244916] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4008.249576] ? watchdog+0x40/0x40 [ 4008.252234] ? hrtimer_interrupt+0x128/0x2c0 [ 4008.255177] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4008.257656] ? apic_timer_interrupt+0xf/0x20 [ 4008.259838] [ 4008.260966] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4008.263626] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4008.270653] do_raw_spin_lock+0x177/0x190 [ 4008.275807] _raw_spin_lock+0x21/0x30 [ 4008.277531] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4008.287454] process_one_work+0x2c8/0x700 [ 4008.288931] worker_thread+0x296/0x6e0 [ 4008.298636] ? rescuer_thread+0x570/0x570 [ 4008.305944] kthread+0x1d1/0x200 [ 4008.307182] ? set_kthread_struct+0x70/0x70 [ 4008.308696] ret_from_fork+0x1f/0x30 [ 4021.002586] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 4021.011923] rcu: 0-....: (1274767 ticks this GP) idle=30a/1/0x4000000000000000 softirq=128519/128519 fqs=126395 [ 4021.028573] (detected by 1, t=2040043 jiffies, g=262441, q=57592) [ 4021.043085] Sending NMI from CPU 1 to CPUs 0: [ 4021.044702] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 4021.049685] rcu: rcu_sched kthread starved for 1509230 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 4021.061745] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4021.074308] rcu: RCU grace-period kthread stack dump: [ 4021.082100] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4021.099406] Call Trace: [ 4021.103377] __schedule+0x351/0xcb0 [ 4021.104704] schedule+0xc0/0x180 [ 4021.105907] schedule_timeout+0xb4/0x190 [ 4021.115676] ? do_raw_spin_unlock+0x75/0x190 [ 4021.120241] ? __next_timer_interrupt+0x160/0x160 [ 4021.122551] ? prepare_to_swait_event+0xf3/0x140 [ 4021.127751] rcu_gp_kthread+0x8dd/0x15b0 [ 4021.129180] ? force_qs_rnp+0x2c0/0x2c0 [ 4021.136550] kthread+0x1d1/0x200 [ 4021.138133] ? set_kthread_struct+0x70/0x70 [ 4021.147505] ret_from_fork+0x1f/0x30 [ 4021.153593] rcu: Stack dump where RCU GP kthread last ran: [ 4021.159892] Sending NMI from CPU 1 to CPUs 3: [ 4021.169529] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [ 4036.060032] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 4036.066883] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4036.123505] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4036.138097] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4036.144912] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4036.149646] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4036.152362] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4036.175206] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4036.185613] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4036.194605] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4036.198416] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4036.206321] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4036.212403] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4036.217623] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4036.223636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4036.228391] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4036.234614] Call Trace: [ 4036.236451] [ 4036.238034] ? show_regs.cold.9+0x22/0x2f [ 4036.240134] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4036.243357] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4036.246086] ? watchdog+0x40/0x40 [ 4036.247984] ? hrtimer_interrupt+0x128/0x2c0 [ 4036.251566] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4036.255882] ? apic_timer_interrupt+0xf/0x20 [ 4036.258584] [ 4036.259714] ? native_safe_halt+0x2e/0x50 [ 4036.264142] kvm_wait+0xcd/0xe0 [ 4036.269124] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4036.274664] do_raw_spin_lock+0x177/0x190 [ 4036.280202] _raw_spin_lock+0x21/0x30 [ 4036.284201] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4036.290620] process_one_work+0x2c8/0x700 [ 4036.291980] worker_thread+0x296/0x6e0 [ 4036.299677] ? rescuer_thread+0x570/0x570 [ 4036.302422] kthread+0x1d1/0x200 [ 4036.306187] ? set_kthread_struct+0x70/0x70 [ 4036.309084] ret_from_fork+0x1f/0x30 [ 4036.576193] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1523s! [ 4036.581011] Showing busy workqueues and worker pools: [ 4036.590647] workqueue events: flags=0x0 [ 4036.600823] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4036.611069] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4036.630275] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4036.641006] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4036.649918] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4036.663240] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4036.672495] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4036.685598] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4036.696075] workqueue events_long: flags=0x0 [ 4036.700287] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4036.710601] pending: lu_objects_flush [obdclass] [ 4036.715738] workqueue events_unbound: flags=0x2 [ 4036.722707] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4036.724789] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4036.734534] workqueue events_power_efficient: flags=0x80 [ 4036.737598] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4036.750896] pending: neigh_periodic_work, do_cache_clean [ 4036.752754] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4036.769941] pending: neigh_periodic_work, check_lifetime [ 4036.774874] workqueue mm_percpu_wq: flags=0x8 [ 4036.776101] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4036.777492] pending: vmstat_update, lru_add_drain_per_cpu [ 4036.784577] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4036.790062] pending: vmstat_update [ 4036.791674] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4036.802459] pending: lru_add_drain_per_cpu BAR(48) [ 4036.806478] workqueue ipv6_addrconf: flags=0x40008 [ 4036.811285] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4036.815164] pending: addrconf_verify_work [ 4036.818258] workqueue knbd0-recv: flags=0x1a [ 4036.822864] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4036.826691] in-flight: 650:recv_work [ 4036.830198] workqueue ptlrpc_pinger: flags=0x2 [ 4036.832656] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4036.835948] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4036.841433] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1523s workers=4 idle: 36 466 29757 [ 4036.852311] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4026s workers=2 idle: 652 [ 4048.059774] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 4048.062021] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 4048.065045] Modules linked in: lustre(O) [ 4048.068159] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4048.068240] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4048.068247] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4048.068250] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4048.068263] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4048.068267] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4048.068271] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4048.068274] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4048.068276] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4048.068278] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4048.068280] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4048.068284] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4048.068286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4048.068288] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4048.068297] Call Trace: [ 4048.068303] [ 4048.068308] ? show_regs.cold.9+0x22/0x2f [ 4048.068318] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4048.068324] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4048.068328] ? watchdog+0x40/0x40 [ 4048.068333] ? hrtimer_interrupt+0x128/0x2c0 [ 4048.068336] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4048.068343] ? apic_timer_interrupt+0xf/0x20 [ 4048.068348] [ 4048.068351] ? native_safe_halt+0x2e/0x50 [ 4048.068355] kvm_wait+0xcd/0xe0 [ 4048.068364] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4048.068371] do_raw_spin_lock+0x177/0x190 [ 4048.068557] _raw_spin_lock+0x21/0x30 [ 4048.068563] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4048.075297] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4048.075369] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4048.098507] ? do_raw_spin_unlock+0x75/0x190 [ 4048.098526] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4048.098802] ? do_wait_intr+0xf0/0xf0 [ 4048.098810] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4048.099163] mdc_reint+0x2d/0x110 [mdc] [ 4048.099213] mdc_unlink+0x658/0x870 [mdc] [ 4048.099241] lmv_unlink+0x34d/0xbd0 [lmv] [ 4048.099272] ll_unlink+0x24d/0x7f0 [lustre] [ 4048.099497] vfs_unlink+0x202/0x330 [ 4048.099505] do_unlinkat+0x2fa/0x460 [ 4048.099511] __x64_sys_unlink+0x37/0x50 [ 4048.099514] do_syscall_64+0xc1/0x440 [ 4048.099521] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4048.099529] RIP: 0033:0x7f83e5f5c10b [ 4048.099536] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4048.099539] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4048.099544] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4048.099547] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4048.099549] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4048.099551] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4048.099554] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4048.274911] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4048.280917] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 4048.284356] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 4048.295172] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4048.299739] RAX: 000000000000778b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4048.303108] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4048.306993] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4048.310756] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4048.314091] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4048.316747] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4048.320998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4048.324345] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4048.328550] Call Trace: [ 4048.329896] [ 4048.331545] ? show_regs.cold.9+0x22/0x2f [ 4048.333299] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4048.336832] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4048.339956] ? watchdog+0x40/0x40 [ 4048.342327] ? hrtimer_interrupt+0x128/0x2c0 [ 4048.344478] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4048.346633] ? apic_timer_interrupt+0xf/0x20 [ 4048.348073] [ 4048.349112] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 4048.351572] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4048.355389] do_raw_spin_lock+0x177/0x190 [ 4048.357539] _raw_spin_lock+0x21/0x30 [ 4048.359845] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4048.363555] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4048.366534] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4048.370208] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4048.373729] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4048.377326] ? woken_wake_function+0x30/0x30 [ 4048.379804] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4048.382830] kthread+0x1d1/0x200 [ 4048.384901] ? set_kthread_struct+0x70/0x70 [ 4048.386923] ret_from_fork+0x1f/0x30 [ 4064.059174] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 4064.062891] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 4064.087644] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4064.087726] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4064.087735] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4064.087737] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4064.087752] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4064.087757] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4064.087762] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4064.087764] RDX: 0000000000006181 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4064.087766] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4064.087769] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4064.087771] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4064.087774] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4064.087777] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4064.087779] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4064.087790] Call Trace: [ 4064.087796] [ 4064.087802] ? show_regs.cold.9+0x22/0x2f [ 4064.087812] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4064.087819] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4064.087823] ? watchdog+0x40/0x40 [ 4064.087828] ? hrtimer_interrupt+0x128/0x2c0 [ 4064.087832] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4064.087841] ? apic_timer_interrupt+0xf/0x20 [ 4064.087847] [ 4064.087850] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4064.087855] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4064.087861] do_raw_spin_lock+0x177/0x190 [ 4064.087866] _raw_spin_lock+0x21/0x30 [ 4064.087872] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4064.088169] obd_mod_free+0x46/0x140 [ptlrpc] [ 4064.088375] mdc_commit_open+0x91/0xb0 [mdc] [ 4064.088418] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4064.088631] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4064.088824] after_reply+0xe2c/0x1780 [ptlrpc] [ 4064.089038] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4064.089257] ? __next_timer_interrupt+0x160/0x160 [ 4064.089269] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4064.089469] ? do_wait_intr+0xf0/0xf0 [ 4064.089477] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4064.089687] kthread+0x1d1/0x200 [ 4064.089692] ? set_kthread_struct+0x70/0x70 [ 4064.089696] ret_from_fork+0x1f/0x30 [ 4064.473508] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4064.535513] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4064.544599] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4064.550546] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4064.561895] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4064.572078] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4064.589216] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4064.591210] RAX: 000000000000312b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4064.597962] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4064.606062] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4064.615375] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4064.620615] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4064.635561] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4064.645205] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4064.657136] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4064.669602] Call Trace: [ 4064.674626] [ 4064.676904] ? show_regs.cold.9+0x22/0x2f [ 4064.682031] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4064.686804] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4064.689474] ? watchdog+0x40/0x40 [ 4064.695962] ? hrtimer_interrupt+0x128/0x2c0 [ 4064.699059] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4064.705886] ? apic_timer_interrupt+0xf/0x20 [ 4064.721775] [ 4064.722621] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4064.740128] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4064.743498] do_raw_spin_lock+0x177/0x190 [ 4064.747966] _raw_spin_lock+0x21/0x30 [ 4064.749386] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4064.752919] process_one_work+0x2c8/0x700 [ 4064.755013] worker_thread+0x296/0x6e0 [ 4064.756936] ? rescuer_thread+0x570/0x570 [ 4064.765674] kthread+0x1d1/0x200 [ 4064.767716] ? set_kthread_struct+0x70/0x70 [ 4064.784942] ret_from_fork+0x1f/0x30 [ 4067.297082] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1554s! [ 4067.302134] Showing busy workqueues and worker pools: [ 4067.304900] workqueue events: flags=0x0 [ 4067.306725] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4067.310259] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4067.314436] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4067.317967] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4067.321688] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4067.325225] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4067.329642] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4067.332849] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4067.337774] workqueue events_long: flags=0x0 [ 4067.339885] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4067.343214] pending: lu_objects_flush [obdclass] [ 4067.346121] workqueue events_unbound: flags=0x2 [ 4067.348688] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4067.352255] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4067.355684] workqueue events_power_efficient: flags=0x80 [ 4067.358297] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4067.362357] pending: neigh_periodic_work, do_cache_clean [ 4067.365987] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4067.370006] pending: neigh_periodic_work, check_lifetime [ 4067.372825] workqueue mm_percpu_wq: flags=0x8 [ 4067.375674] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4067.378772] pending: vmstat_update, lru_add_drain_per_cpu [ 4067.381169] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4067.384141] pending: vmstat_update [ 4067.386052] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4067.389842] pending: lru_add_drain_per_cpu BAR(48) [ 4067.394084] workqueue ipv6_addrconf: flags=0x40008 [ 4067.397316] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4067.401424] pending: addrconf_verify_work [ 4067.403967] workqueue knbd0-recv: flags=0x1a [ 4067.405915] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4067.409587] in-flight: 650:recv_work [ 4067.412023] workqueue ptlrpc_pinger: flags=0x2 [ 4067.414106] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4067.417021] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4067.419870] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1554s workers=4 idle: 36 466 29757 [ 4067.423654] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4057s workers=2 idle: 652 [ 4076.059387] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 4076.062395] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 4076.076588] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [ 4076.081216] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4076.081370] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4076.081377] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4076.081380] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4076.081394] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4076.081399] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4076.081403] RAX: 0000000000002d07 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4076.081406] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4076.081408] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4076.081410] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4076.081423] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4076.081427] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4076.081430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4076.081433] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4076.081441] Call Trace: [ 4076.081446] [ 4076.081453] ? show_regs.cold.9+0x22/0x2f [ 4076.081462] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4076.081469] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4076.081474] ? watchdog+0x40/0x40 [ 4076.081478] ? hrtimer_interrupt+0x128/0x2c0 [ 4076.081482] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4076.081490] ? apic_timer_interrupt+0xf/0x20 [ 4076.081494] [ 4076.081497] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4076.081502] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4076.081507] do_raw_spin_lock+0x177/0x190 [ 4076.081512] _raw_spin_lock+0x21/0x30 [ 4076.081517] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4076.082079] ? do_raw_spin_unlock+0x75/0x190 [ 4076.082085] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4076.082366] ? do_wait_intr+0xf0/0xf0 [ 4076.082374] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4076.082584] mdc_reint+0x2d/0x110 [mdc] [ 4076.082892] mdc_unlink+0x658/0x870 [mdc] [ 4076.082922] lmv_unlink+0x34d/0xbd0 [lmv] [ 4076.082964] ll_unlink+0x24d/0x7f0 [lustre] [ 4076.083106] vfs_unlink+0x202/0x330 [ 4076.083114] do_unlinkat+0x2fa/0x460 [ 4076.083119] __x64_sys_unlink+0x37/0x50 [ 4076.083122] do_syscall_64+0xc1/0x440 [ 4076.083129] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4076.083136] RIP: 0033:0x7f83e5f5c10b [ 4076.083143] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4076.083146] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4076.083151] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4076.083153] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4076.083155] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4076.083157] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4076.083159] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4076.313723] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4076.338506] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4076.346487] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4076.352504] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4076.355040] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4076.366808] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4076.371052] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4076.375254] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4076.379066] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4076.383033] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4076.387615] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4076.392070] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4076.398404] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4076.402228] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4076.407061] Call Trace: [ 4076.409002] [ 4076.410083] ? show_regs.cold.9+0x22/0x2f [ 4076.413440] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4076.416286] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4076.419099] ? watchdog+0x40/0x40 [ 4076.421081] ? hrtimer_interrupt+0x128/0x2c0 [ 4076.423849] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4076.426763] ? apic_timer_interrupt+0xf/0x20 [ 4076.429290] [ 4076.431312] ? native_safe_halt+0x2e/0x50 [ 4076.433175] kvm_wait+0xcd/0xe0 [ 4076.435587] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4076.438544] do_raw_spin_lock+0x177/0x190 [ 4076.441199] _raw_spin_lock+0x21/0x30 [ 4076.443529] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4076.447770] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4076.451394] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4076.454474] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4076.457969] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4076.462491] ? woken_wake_function+0x30/0x30 [ 4076.465560] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4076.469171] kthread+0x1d1/0x200 [ 4076.471187] ? set_kthread_struct+0x70/0x70 [ 4076.473621] ret_from_fork+0x1f/0x30 [ 4092.060645] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 4092.061020] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 4092.065611] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4092.065704] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4092.065710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4092.065716] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4092.066098] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4092.066109] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4092.066113] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4092.066117] RAX: 0000000000007d26 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4092.066120] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4092.066123] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4092.066125] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4092.066128] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4092.066131] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4092.066134] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4092.066136] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4092.066146] Call Trace: [ 4092.066152] [ 4092.066157] ? show_regs.cold.9+0x22/0x2f [ 4092.066166] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4092.066172] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4092.066176] ? watchdog+0x40/0x40 [ 4092.066181] ? hrtimer_interrupt+0x128/0x2c0 [ 4092.066185] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4092.066192] ? apic_timer_interrupt+0xf/0x20 [ 4092.066198] [ 4092.066201] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4092.066206] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4092.066211] do_raw_spin_lock+0x177/0x190 [ 4092.066216] _raw_spin_lock+0x21/0x30 [ 4092.066221] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4092.066433] process_one_work+0x2c8/0x700 [ 4092.066441] worker_thread+0x296/0x6e0 [ 4092.066446] ? rescuer_thread+0x570/0x570 [ 4092.066449] kthread+0x1d1/0x200 [ 4092.066454] ? set_kthread_struct+0x70/0x70 [ 4092.066458] ret_from_fork+0x1f/0x30 [ 4092.378833] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4092.433027] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4092.439112] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4092.454958] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4092.463846] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4092.486091] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4092.490899] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4092.493344] RDX: 000000000000001e RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4092.499937] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4092.510450] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4092.528406] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4092.537866] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4092.555588] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4092.567059] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4092.571128] Call Trace: [ 4092.575922] [ 4092.576658] ? show_regs.cold.9+0x22/0x2f [ 4092.585001] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4092.588645] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4092.592910] ? watchdog+0x40/0x40 [ 4092.596829] ? hrtimer_interrupt+0x128/0x2c0 [ 4092.604985] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4092.609095] ? apic_timer_interrupt+0xf/0x20 [ 4092.614125] [ 4092.615277] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4092.625099] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4092.627165] do_raw_spin_lock+0x177/0x190 [ 4092.634042] _raw_spin_lock+0x21/0x30 [ 4092.635127] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4092.639756] obd_mod_free+0x46/0x140 [ptlrpc] [ 4092.644857] mdc_commit_open+0x91/0xb0 [mdc] [ 4092.653345] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4092.655950] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4092.665961] after_reply+0xe2c/0x1780 [ptlrpc] [ 4092.675549] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4092.679626] ? __next_timer_interrupt+0x160/0x160 [ 4092.687779] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4092.695345] ? do_wait_intr+0xf0/0xf0 [ 4092.696944] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4092.700549] kthread+0x1d1/0x200 [ 4092.708126] ? set_kthread_struct+0x70/0x70 [ 4092.714401] ret_from_fork+0x1f/0x30 [ 4097.504426] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1584s! [ 4097.508585] Showing busy workqueues and worker pools: [ 4097.511348] workqueue events: flags=0x0 [ 4097.512853] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4097.515830] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4097.519386] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4097.522025] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4097.525393] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4097.528910] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4097.533647] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4097.537495] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4097.542137] workqueue events_long: flags=0x0 [ 4097.544472] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4097.548557] pending: lu_objects_flush [obdclass] [ 4097.550831] workqueue events_unbound: flags=0x2 [ 4097.553396] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4097.555960] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4097.563756] workqueue events_power_efficient: flags=0x80 [ 4097.566651] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4097.573303] pending: neigh_periodic_work, do_cache_clean [ 4097.577388] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4097.580936] pending: neigh_periodic_work, check_lifetime [ 4097.583723] workqueue mm_percpu_wq: flags=0x8 [ 4097.587030] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4097.592165] pending: vmstat_update, lru_add_drain_per_cpu [ 4097.595096] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4097.600802] pending: vmstat_update [ 4097.605939] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4097.610761] pending: lru_add_drain_per_cpu BAR(48) [ 4097.613895] workqueue ipv6_addrconf: flags=0x40008 [ 4097.617126] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4097.621530] pending: addrconf_verify_work [ 4097.623726] workqueue knbd0-recv: flags=0x1a [ 4097.626124] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4097.629794] in-flight: 650:recv_work [ 4097.632013] workqueue ptlrpc_pinger: flags=0x2 [ 4097.635039] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4097.637749] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4097.642835] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1584s workers=4 idle: 36 466 29757 [ 4097.649138] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4087s workers=2 idle: 652 [ 4104.062223] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 4104.063293] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 4104.068539] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4104.068616] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4104.068845] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4104.068848] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4104.068865] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4104.068870] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4104.068875] RAX: 0000000000007e2f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4104.068878] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4104.068880] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4104.068882] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4104.068884] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4104.068887] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4104.068891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4104.068893] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4104.068904] Call Trace: [ 4104.068910] [ 4104.068915] ? show_regs.cold.9+0x22/0x2f [ 4104.068926] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4104.068934] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4104.068939] ? watchdog+0x40/0x40 [ 4104.068944] ? hrtimer_interrupt+0x128/0x2c0 [ 4104.068948] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4104.068957] ? apic_timer_interrupt+0xf/0x20 [ 4104.068962] [ 4104.068965] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4104.068970] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4104.068976] do_raw_spin_lock+0x177/0x190 [ 4104.068986] _raw_spin_lock+0x21/0x30 [ 4104.068991] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4104.069472] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4104.069849] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4104.078744] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4104.078826] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4104.078834] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4104.078837] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4104.078852] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4104.078857] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4104.078862] RAX: 000000000000668a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4104.078864] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4104.078866] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4104.078868] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4104.078871] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4104.078874] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4104.078878] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4104.078880] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4104.078888] Call Trace: [ 4104.078894] [ 4104.078899] ? show_regs.cold.9+0x22/0x2f [ 4104.078909] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4104.078916] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4104.078920] ? watchdog+0x40/0x40 [ 4104.078924] ? hrtimer_interrupt+0x128/0x2c0 [ 4104.078928] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4104.078937] ? apic_timer_interrupt+0xf/0x20 [ 4104.078942] [ 4104.078945] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4104.078950] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4104.078954] do_raw_spin_lock+0x177/0x190 [ 4104.078959] _raw_spin_lock+0x21/0x30 [ 4104.078965] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4104.079278] ? do_raw_spin_unlock+0x75/0x190 [ 4104.079285] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4104.079515] ? do_wait_intr+0xf0/0xf0 [ 4104.079528] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4104.079999] mdc_reint+0x2d/0x110 [mdc] [ 4104.080056] mdc_unlink+0x658/0x870 [mdc] [ 4104.080092] lmv_unlink+0x34d/0xbd0 [lmv] [ 4104.080125] ll_unlink+0x24d/0x7f0 [lustre] [ 4104.080257] vfs_unlink+0x202/0x330 [ 4104.080265] do_unlinkat+0x2fa/0x460 [ 4104.080276] __x64_sys_unlink+0x37/0x50 [ 4104.080280] do_syscall_64+0xc1/0x440 [ 4104.080288] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4104.080296] RIP: 0033:0x7f83e5f5c10b [ 4104.080306] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4104.080310] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4104.080315] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4104.080317] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4104.080320] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4104.080322] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4104.080324] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4104.327869] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4104.330657] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4104.333440] ? woken_wake_function+0x30/0x30 [ 4104.335683] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4104.338514] kthread+0x1d1/0x200 [ 4104.339992] ? set_kthread_struct+0x70/0x70 [ 4104.342368] ret_from_fork+0x1f/0x30 [ 4120.062417] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 4120.067697] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4120.090816] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4120.097030] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4120.101754] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4120.106159] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4120.110003] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4120.124165] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4120.128419] RAX: 00000000000038b8 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4120.132383] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4120.136193] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4120.140474] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4120.144654] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4120.148598] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4120.152652] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4120.156147] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4120.160076] Call Trace: [ 4120.162200] [ 4120.163430] ? show_regs.cold.9+0x22/0x2f [ 4120.165482] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4120.167625] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4120.169865] ? watchdog+0x40/0x40 [ 4120.171510] ? hrtimer_interrupt+0x128/0x2c0 [ 4120.173437] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4120.175830] ? apic_timer_interrupt+0xf/0x20 [ 4120.177492] [ 4120.178273] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4120.181942] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4120.184488] do_raw_spin_lock+0x177/0x190 [ 4120.186140] _raw_spin_lock+0x21/0x30 [ 4120.187846] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4120.190476] process_one_work+0x2c8/0x700 [ 4120.192143] worker_thread+0x296/0x6e0 [ 4120.194100] ? rescuer_thread+0x570/0x570 [ 4120.196761] kthread+0x1d1/0x200 [ 4120.198507] ? set_kthread_struct+0x70/0x70 [ 4120.200677] ret_from_fork+0x1f/0x30 [ 4124.057205] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 4124.063451] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4124.101073] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4124.111639] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4124.125493] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4124.127901] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4124.140870] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4124.148276] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4124.151644] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4124.155109] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4124.162059] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4124.164426] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4124.171316] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4124.181768] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4124.185607] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4124.189668] Call Trace: [ 4124.191207] [ 4124.191937] ? show_regs.cold.9+0x22/0x2f [ 4124.193976] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4124.195490] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4124.197508] ? watchdog+0x40/0x40 [ 4124.199400] ? hrtimer_interrupt+0x128/0x2c0 [ 4124.201125] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4124.203088] ? apic_timer_interrupt+0xf/0x20 [ 4124.204759] [ 4124.207433] ? native_safe_halt+0x2e/0x50 [ 4124.209303] kvm_wait+0xcd/0xe0 [ 4124.210846] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4124.212638] do_raw_spin_lock+0x177/0x190 [ 4124.216832] _raw_spin_lock+0x21/0x30 [ 4124.218611] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4124.222670] obd_mod_free+0x46/0x140 [ptlrpc] [ 4124.226409] mdc_commit_open+0x91/0xb0 [mdc] [ 4124.228680] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4124.232940] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4124.238903] after_reply+0xe2c/0x1780 [ptlrpc] [ 4124.241302] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4124.243555] ? __next_timer_interrupt+0x160/0x160 [ 4124.246276] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4124.249688] ? do_wait_intr+0xf0/0xf0 [ 4124.253483] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4124.255791] kthread+0x1d1/0x200 [ 4124.260746] ? set_kthread_struct+0x70/0x70 [ 4124.264634] ret_from_fork+0x1f/0x30 [ 4127.714030] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1614s! [ 4127.729052] Showing busy workqueues and worker pools: [ 4127.732862] workqueue events: flags=0x0 [ 4127.736264] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4127.743406] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4127.751807] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4127.755077] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4127.763308] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4127.769794] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4127.777947] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4127.783690] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4127.792013] workqueue events_long: flags=0x0 [ 4127.793991] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4127.797519] pending: lu_objects_flush [obdclass] [ 4127.800332] workqueue events_unbound: flags=0x2 [ 4127.802022] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4127.805285] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4127.807919] workqueue events_power_efficient: flags=0x80 [ 4127.813866] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4127.817632] pending: neigh_periodic_work, do_cache_clean [ 4127.820921] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4127.828307] pending: neigh_periodic_work, check_lifetime [ 4127.841847] workqueue mm_percpu_wq: flags=0x8 [ 4127.843316] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4127.854937] pending: vmstat_update, lru_add_drain_per_cpu [ 4127.867594] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4127.875224] pending: vmstat_update [ 4127.886434] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4127.893573] pending: lru_add_drain_per_cpu BAR(48) [ 4127.895044] workqueue ipv6_addrconf: flags=0x40008 [ 4127.897294] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4127.900519] pending: addrconf_verify_work [ 4127.904611] workqueue knbd0-recv: flags=0x1a [ 4127.906861] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4127.909626] in-flight: 650:recv_work [ 4127.910902] workqueue ptlrpc_pinger: flags=0x2 [ 4127.912874] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4127.914977] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4127.917226] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1614s workers=4 idle: 36 466 29757 [ 4127.919919] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4117s workers=2 idle: 652 [ 4132.059028] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 4132.062740] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 4132.074578] Modules linked in: [ 4132.079444] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4132.079518] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4132.079525] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4132.079527] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4132.079541] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4132.079545] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4132.079549] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4132.079551] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4132.079553] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4132.079555] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4132.079557] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4132.079560] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4132.079563] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4132.079565] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4132.079574] Call Trace: [ 4132.079580] [ 4132.079591] ? show_regs.cold.9+0x22/0x2f [ 4132.079601] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4132.079607] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4132.079612] ? watchdog+0x40/0x40 [ 4132.079616] ? hrtimer_interrupt+0x128/0x2c0 [ 4132.079620] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4132.079626] ? apic_timer_interrupt+0xf/0x20 [ 4132.079631] [ 4132.079634] ? native_safe_halt+0x2e/0x50 [ 4132.079637] kvm_wait+0xcd/0xe0 [ 4132.079645] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4132.079652] do_raw_spin_lock+0x177/0x190 [ 4132.079656] _raw_spin_lock+0x21/0x30 [ 4132.079673] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4132.080201] ? do_raw_spin_unlock+0x75/0x190 [ 4132.080208] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4132.080594] ? do_wait_intr+0xf0/0xf0 [ 4132.080603] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4132.080830] mdc_reint+0x2d/0x110 [mdc] [ 4132.081046] mdc_unlink+0x658/0x870 [mdc] [ 4132.081082] lmv_unlink+0x34d/0xbd0 [lmv] [ 4132.081114] ll_unlink+0x24d/0x7f0 [lustre] [ 4132.081227] vfs_unlink+0x202/0x330 [ 4132.081235] do_unlinkat+0x2fa/0x460 [ 4132.081240] __x64_sys_unlink+0x37/0x50 [ 4132.081244] do_syscall_64+0xc1/0x440 [ 4132.081251] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4132.081258] RIP: 0033:0x7f83e5f5c10b [ 4132.081264] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4132.081268] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4132.081273] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4132.081276] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4132.081277] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4132.081280] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4132.081282] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4132.270099] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4132.298391] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4132.304154] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4132.312139] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4132.317501] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4132.327712] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4132.330778] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4132.334586] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4132.337727] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4132.340644] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4132.346288] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4132.350124] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4132.353831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4132.356773] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4132.360006] Call Trace: [ 4132.361215] [ 4132.362170] ? show_regs.cold.9+0x22/0x2f [ 4132.363808] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4132.366160] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4132.368553] ? watchdog+0x40/0x40 [ 4132.370444] ? hrtimer_interrupt+0x128/0x2c0 [ 4132.373212] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4132.380916] ? apic_timer_interrupt+0xf/0x20 [ 4132.386433] [ 4132.387405] ? native_safe_halt+0x2e/0x50 [ 4132.391251] kvm_wait+0xcd/0xe0 [ 4132.392507] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4132.396715] do_raw_spin_lock+0x177/0x190 [ 4132.398494] _raw_spin_lock+0x21/0x30 [ 4132.400541] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4132.404239] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4132.406572] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4132.408160] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4132.410182] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4132.412674] ? woken_wake_function+0x30/0x30 [ 4132.415046] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4132.417906] kthread+0x1d1/0x200 [ 4132.419337] ? set_kthread_struct+0x70/0x70 [ 4132.421686] ret_from_fork+0x1f/0x30 [ 4148.060046] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 4148.062670] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4148.084228] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4148.091349] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4148.095933] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4148.098733] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4148.103296] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4148.112858] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4148.118520] RAX: 000000000000328b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4148.124878] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4148.131828] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4148.139938] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4148.145447] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4148.152196] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4148.155324] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4148.158907] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4148.163568] Call Trace: [ 4148.168335] [ 4148.172404] ? show_regs.cold.9+0x22/0x2f [ 4148.177447] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4148.180168] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4148.182885] ? watchdog+0x40/0x40 [ 4148.184511] ? hrtimer_interrupt+0x128/0x2c0 [ 4148.194008] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4148.195578] ? apic_timer_interrupt+0xf/0x20 [ 4148.207429] [ 4148.209545] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4148.215283] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4148.219046] do_raw_spin_lock+0x177/0x190 [ 4148.224052] _raw_spin_lock+0x21/0x30 [ 4148.229079] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4148.231489] process_one_work+0x2c8/0x700 [ 4148.233559] worker_thread+0x296/0x6e0 [ 4148.236422] ? rescuer_thread+0x570/0x570 [ 4148.242849] kthread+0x1d1/0x200 [ 4148.243941] ? set_kthread_struct+0x70/0x70 [ 4148.248199] ret_from_fork+0x1f/0x30 [ 4152.056195] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 4152.063168] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4152.100200] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4152.107705] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4152.117325] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4152.118727] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4152.127044] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4152.130458] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4152.133009] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4152.136942] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4152.139263] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4152.144458] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4152.149364] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4152.153041] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4152.156951] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4152.161298] Call Trace: [ 4152.164222] [ 4152.165397] ? show_regs.cold.9+0x22/0x2f [ 4152.167763] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4152.171890] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4152.177986] ? watchdog+0x40/0x40 [ 4152.182442] ? hrtimer_interrupt+0x128/0x2c0 [ 4152.188926] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4152.195817] ? apic_timer_interrupt+0xf/0x20 [ 4152.200018] [ 4152.202380] ? native_safe_halt+0x2e/0x50 [ 4152.206059] kvm_wait+0xcd/0xe0 [ 4152.208441] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4152.217311] do_raw_spin_lock+0x177/0x190 [ 4152.220429] _raw_spin_lock+0x21/0x30 [ 4152.228647] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4152.239100] obd_mod_free+0x46/0x140 [ptlrpc] [ 4152.246524] mdc_commit_open+0x91/0xb0 [mdc] [ 4152.252174] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4152.257855] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4152.265864] after_reply+0xe2c/0x1780 [ptlrpc] [ 4152.269007] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4152.276232] ? __next_timer_interrupt+0x160/0x160 [ 4152.278035] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4152.282227] ? do_wait_intr+0xf0/0xf0 [ 4152.289464] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4152.297614] kthread+0x1d1/0x200 [ 4152.301966] ? set_kthread_struct+0x70/0x70 [ 4152.304344] ret_from_fork+0x1f/0x30 [ 4158.432297] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1645s! [ 4158.437058] Showing busy workqueues and worker pools: [ 4158.439114] workqueue events: flags=0x0 [ 4158.440474] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4158.442784] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4158.446951] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4158.449930] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4158.452678] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4158.455152] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4158.458826] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4158.461998] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4158.466169] workqueue events_long: flags=0x0 [ 4158.468872] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4158.474038] pending: lu_objects_flush [obdclass] [ 4158.476927] workqueue events_unbound: flags=0x2 [ 4158.478993] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4158.481234] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4158.484016] workqueue events_power_efficient: flags=0x80 [ 4158.486129] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4158.489105] pending: neigh_periodic_work, do_cache_clean [ 4158.491265] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4158.494076] pending: neigh_periodic_work, check_lifetime [ 4158.496553] workqueue mm_percpu_wq: flags=0x8 [ 4158.498604] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4158.501140] pending: vmstat_update, lru_add_drain_per_cpu [ 4158.503569] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4158.506534] pending: vmstat_update [ 4158.508119] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4158.510772] pending: lru_add_drain_per_cpu BAR(48) [ 4158.513397] workqueue ipv6_addrconf: flags=0x40008 [ 4158.515502] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4158.518267] pending: addrconf_verify_work [ 4158.519941] workqueue knbd0-recv: flags=0x1a [ 4158.521383] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4158.524012] in-flight: 650:recv_work [ 4158.525475] workqueue ptlrpc_pinger: flags=0x2 [ 4158.527061] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4158.529619] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4158.531889] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1645s workers=4 idle: 36 466 29757 [ 4158.535053] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4148s workers=2 idle: 652 [ 4160.059236] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 4160.062048] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4160.063189] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4160.063262] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4160.063269] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4160.063272] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4160.067809] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) [ 4160.105527] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4160.111558] fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4160.111621] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4160.111628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4160.111631] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4160.111645] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4160.111649] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4160.111653] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4160.111655] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4160.111657] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4160.111659] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4160.111662] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4160.111665] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4160.111667] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4160.111669] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4160.111677] Call Trace: [ 4160.111682] [ 4160.111689] ? show_regs.cold.9+0x22/0x2f [ 4160.111709] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4160.111720] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4160.111726] ? watchdog+0x40/0x40 [ 4160.111733] ? hrtimer_interrupt+0x128/0x2c0 [ 4160.111736] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4160.111742] ? apic_timer_interrupt+0xf/0x20 [ 4160.111746] [ 4160.111748] ? native_safe_halt+0x2e/0x50 [ 4160.111752] kvm_wait+0xcd/0xe0 [ 4160.111759] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4160.111766] do_raw_spin_lock+0x177/0x190 [ 4160.111770] _raw_spin_lock+0x21/0x30 [ 4160.111775] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4160.112367] ? do_raw_spin_unlock+0x75/0x190 [ 4160.112373] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4160.112702] ? do_wait_intr+0xf0/0xf0 [ 4160.112711] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4160.117182] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4160.117196] RAX: 0000000000006285 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4160.117199] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4160.117201] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4160.117203] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4160.117206] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4160.117209] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4160.117212] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4160.117214] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4160.117223] Call Trace: [ 4160.117234] [ 4160.117239] ? show_regs.cold.9+0x22/0x2f [ 4160.117253] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4160.117259] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4160.117263] ? watchdog+0x40/0x40 [ 4160.117268] ? hrtimer_interrupt+0x128/0x2c0 [ 4160.117272] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4160.117280] ? apic_timer_interrupt+0xf/0x20 [ 4160.117284] [ 4160.117287] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4160.117293] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4160.117301] do_raw_spin_lock+0x177/0x190 [ 4160.117305] _raw_spin_lock+0x21/0x30 [ 4160.117309] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4160.117828] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4160.118176] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4160.118400] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4160.118648] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4160.119009] ? woken_wake_function+0x30/0x30 [ 4160.119019] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4160.119233] kthread+0x1d1/0x200 [ 4160.119239] ? set_kthread_struct+0x70/0x70 [ 4160.119243] ret_from_fork+0x1f/0x30 [ 4160.343194] mdc_reint+0x2d/0x110 [mdc] [ 4160.345199] mdc_unlink+0x658/0x870 [mdc] [ 4160.347151] lmv_unlink+0x34d/0xbd0 [lmv] [ 4160.349120] ll_unlink+0x24d/0x7f0 [lustre] [ 4160.350958] vfs_unlink+0x202/0x330 [ 4160.353146] do_unlinkat+0x2fa/0x460 [ 4160.355122] __x64_sys_unlink+0x37/0x50 [ 4160.356940] do_syscall_64+0xc1/0x440 [ 4160.358577] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4160.361086] RIP: 0033:0x7f83e5f5c10b [ 4160.362699] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4160.372361] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4160.375990] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4160.379446] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4160.383290] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4160.386749] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4160.390476] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4176.060008] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 4176.065874] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4176.115240] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4176.120141] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4176.128578] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4176.135345] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 4176.142569] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 4176.164565] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4176.171824] RAX: 0000000000007f1f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4176.174390] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4176.180543] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4176.187561] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4176.189682] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4176.208632] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4176.214661] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4176.222047] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4176.225211] Call Trace: [ 4176.225926] [ 4176.230328] ? show_regs.cold.9+0x22/0x2f [ 4176.235604] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4176.239612] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4176.241183] ? watchdog+0x40/0x40 [ 4176.242157] ? hrtimer_interrupt+0x128/0x2c0 [ 4176.244691] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4176.247064] ? apic_timer_interrupt+0xf/0x20 [ 4176.250692] [ 4176.251704] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 4176.254423] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4176.256373] do_raw_spin_lock+0x177/0x190 [ 4176.258143] _raw_spin_lock+0x21/0x30 [ 4176.259175] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4176.267567] process_one_work+0x2c8/0x700 [ 4176.268764] worker_thread+0x296/0x6e0 [ 4176.273671] ? rescuer_thread+0x570/0x570 [ 4176.275252] kthread+0x1d1/0x200 [ 4176.279778] ? set_kthread_struct+0x70/0x70 [ 4176.282141] ret_from_fork+0x1f/0x30 [ 4184.061730] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 4184.070139] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4184.111699] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4184.117268] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4184.124893] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4184.127811] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4184.139728] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4184.143667] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4184.147914] RDX: 0000000000006eac RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4184.152936] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4184.155261] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4184.176079] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4184.188998] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4184.199975] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4184.209195] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4184.218949] Call Trace: [ 4184.220618] [ 4184.222859] ? show_regs.cold.9+0x22/0x2f [ 4184.229603] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4184.232720] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4184.234979] ? watchdog+0x40/0x40 [ 4184.236642] ? hrtimer_interrupt+0x128/0x2c0 [ 4184.238906] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4184.241256] ? apic_timer_interrupt+0xf/0x20 [ 4184.243176] [ 4184.251102] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4184.259855] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4184.263710] do_raw_spin_lock+0x177/0x190 [ 4184.266239] _raw_spin_lock+0x21/0x30 [ 4184.268709] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4184.284069] obd_mod_free+0x46/0x140 [ptlrpc] [ 4184.294812] mdc_commit_open+0x91/0xb0 [mdc] [ 4184.297427] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4184.312043] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4184.330836] after_reply+0xe2c/0x1780 [ptlrpc] [ 4184.337509] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4184.345375] ? __next_timer_interrupt+0x160/0x160 [ 4184.352898] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4184.359343] ? do_wait_intr+0xf0/0xf0 [ 4184.366019] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4184.371758] kthread+0x1d1/0x200 [ 4184.375040] ? set_kthread_struct+0x70/0x70 [ 4184.376538] ret_from_fork+0x1f/0x30 [ 4188.061721] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 4188.070279] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4188.076302] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4188.076413] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4188.076420] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4188.076422] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4188.076437] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4188.076442] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4188.076446] RAX: 0000000000006e8f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4188.076448] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4188.076450] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4188.076452] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4188.076454] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4188.076457] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4188.076459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4188.076462] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4188.076470] Call Trace: [ 4188.076475] [ 4188.076479] ? show_regs.cold.9+0x22/0x2f [ 4188.076488] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4188.076494] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4188.076498] ? watchdog+0x40/0x40 [ 4188.076502] ? hrtimer_interrupt+0x128/0x2c0 [ 4188.076506] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4188.076513] ? apic_timer_interrupt+0xf/0x20 [ 4188.076517] [ 4188.076520] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4188.076524] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4188.076530] do_raw_spin_lock+0x177/0x190 [ 4188.076534] _raw_spin_lock+0x21/0x30 [ 4188.076538] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4188.076760] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4188.076911] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4188.077124] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4188.077352] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4188.077567] ? woken_wake_function+0x30/0x30 [ 4188.077576] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4188.088661] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4188.088739] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4188.088746] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4188.088749] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4188.088761] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4188.088766] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4188.088770] RAX: 0000000000007a52 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4188.088773] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4188.088775] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4188.088777] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4188.088779] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4188.088782] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4188.088784] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4188.088786] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4188.088795] Call Trace: [ 4188.088801] [ 4188.088807] ? show_regs.cold.9+0x22/0x2f [ 4188.088817] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4188.088824] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4188.088828] ? watchdog+0x40/0x40 [ 4188.088845] ? hrtimer_interrupt+0x128/0x2c0 [ 4188.088850] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4188.088858] ? apic_timer_interrupt+0xf/0x20 [ 4188.088862] [ 4188.088865] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4188.088869] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4188.088874] do_raw_spin_lock+0x177/0x190 [ 4188.088878] _raw_spin_lock+0x21/0x30 [ 4188.088884] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4188.092552] ? do_raw_spin_unlock+0x75/0x190 [ 4188.125654] kthread+0x1d1/0x200 [ 4188.125673] ? set_kthread_struct+0x70/0x70 [ 4188.125678] ret_from_fork+0x1f/0x30 [ 4188.406844] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4188.409098] ? do_wait_intr+0xf0/0xf0 [ 4188.410718] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4188.413155] mdc_reint+0x2d/0x110 [mdc] [ 4188.414575] mdc_unlink+0x658/0x870 [mdc] [ 4188.416008] lmv_unlink+0x34d/0xbd0 [lmv] [ 4188.417604] ll_unlink+0x24d/0x7f0 [lustre] [ 4188.419638] vfs_unlink+0x202/0x330 [ 4188.420925] do_unlinkat+0x2fa/0x460 [ 4188.422184] __x64_sys_unlink+0x37/0x50 [ 4188.423627] do_syscall_64+0xc1/0x440 [ 4188.424914] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4188.427430] RIP: 0033:0x7f83e5f5c10b [ 4188.428789] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4188.438400] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4188.441803] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4188.445146] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4188.450888] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4188.459864] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4188.467768] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4188.641048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1675s! [ 4188.644465] Showing busy workqueues and worker pools: [ 4188.646600] workqueue events: flags=0x0 [ 4188.647983] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4188.650927] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4188.654976] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4188.658670] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4188.661595] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4188.664742] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4188.668910] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4188.672492] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4188.676965] workqueue events_long: flags=0x0 [ 4188.679442] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4188.683729] pending: lu_objects_flush [obdclass] [ 4188.686451] workqueue events_unbound: flags=0x2 [ 4188.688584] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4188.691606] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4188.694672] workqueue events_power_efficient: flags=0x80 [ 4188.696873] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4188.699706] pending: neigh_periodic_work, do_cache_clean [ 4188.702346] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4188.705278] pending: neigh_periodic_work, check_lifetime [ 4188.707781] workqueue mm_percpu_wq: flags=0x8 [ 4188.710115] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4188.713632] pending: vmstat_update, lru_add_drain_per_cpu [ 4188.716674] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4188.719584] pending: vmstat_update [ 4188.721700] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4188.724668] pending: lru_add_drain_per_cpu BAR(48) [ 4188.726848] workqueue ipv6_addrconf: flags=0x40008 [ 4188.734360] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4188.740950] pending: addrconf_verify_work [ 4188.744596] workqueue knbd0-recv: flags=0x1a [ 4188.745930] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4188.748952] in-flight: 650:recv_work [ 4188.751514] workqueue ptlrpc_pinger: flags=0x2 [ 4188.753816] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4188.756787] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4188.759906] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1675s workers=4 idle: 36 466 29757 [ 4188.763810] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4178s workers=2 idle: 652 [ 4201.007236] rcu: INFO: rcu_sched self-detected stall on CPU [ 4201.010573] rcu: 0-....: (1388943 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 4201.018044] (t=2220052 jiffies g=262441 q=57669) [ 4201.024183] rcu: rcu_sched kthread starved for 1689250 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 4201.036007] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4201.042081] rcu: RCU grace-period kthread stack dump: [ 4201.050273] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4201.064758] Call Trace: [ 4201.067464] __schedule+0x351/0xcb0 [ 4201.072843] schedule+0xc0/0x180 [ 4201.080974] schedule_timeout+0xb4/0x190 [ 4201.084008] ? do_raw_spin_unlock+0x75/0x190 [ 4201.088574] ? __next_timer_interrupt+0x160/0x160 [ 4201.094207] ? prepare_to_swait_event+0xf3/0x140 [ 4201.096540] rcu_gp_kthread+0x8dd/0x15b0 [ 4201.101223] ? force_qs_rnp+0x2c0/0x2c0 [ 4201.103016] kthread+0x1d1/0x200 [ 4201.105067] ? set_kthread_struct+0x70/0x70 [ 4201.107356] ret_from_fork+0x1f/0x30 [ 4201.109240] rcu: Stack dump where RCU GP kthread last ran: [ 4201.113116] Sending NMI from CPU 0 to CPUs 1: [ 4201.121410] NMI backtrace for cpu 1 [ 4201.121417] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4201.121419] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4201.121622] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4201.121624] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4201.121626] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 [ 4201.121629] RAX: 0000000000006602 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4201.121631] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4201.121632] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4201.121633] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4201.121635] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4201.121636] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4201.121637] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4201.121639] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4201.121640] Call Trace: [ 4201.121641] [ 4201.121643] ? show_regs.cold.9+0x22/0x2f [ 4201.121644] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4201.121645] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4201.121646] ? nmi_handle+0xbd/0x210 [ 4201.121647] ? default_do_nmi+0x4b/0x1b0 [ 4201.121648] ? do_nmi+0x280/0x380 [ 4201.121649] ? end_repeat_nmi+0x16/0x1f [ 4201.121650] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4201.121651] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4201.121653] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4201.121654] [ 4201.121655] do_raw_spin_lock+0x177/0x190 [ 4201.121656] _raw_spin_lock+0x21/0x30 [ 4201.121657] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4201.121659] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4201.121660] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4201.121661] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4201.121662] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4201.121663] ? woken_wake_function+0x30/0x30 [ 4201.121664] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4201.121666] kthread+0x1d1/0x200 [ 4201.121667] ? set_kthread_struct+0x70/0x70 [ 4201.121668] ret_from_fork+0x1f/0x30 [ 4201.125398] NMI backtrace for cpu 0 [ 4201.276656] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4201.280382] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4201.283676] Call Trace: [ 4201.284610] [ 4201.285471] ? dump_stack+0xbb/0x10e [ 4201.286717] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4201.288328] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4201.289923] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4201.291352] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4201.293186] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4201.295016] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4201.296527] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4201.310182] ? update_process_times+0x6d/0xc0 [ 4201.316953] ? tick_sched_handle.isra.18+0x27/0x90 [ 4201.325312] ? tick_sched_timer+0x53/0xb0 [ 4201.328785] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4201.336114] ? tick_sched_handle.isra.18+0x90/0x90 [ 4201.340365] ? hrtimer_interrupt+0x128/0x2c0 [ 4201.345984] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4201.349158] ? apic_timer_interrupt+0xf/0x20 [ 4201.355102] [ 4201.356097] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4201.363221] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4201.369297] ? do_raw_spin_lock+0x177/0x190 [ 4201.371245] ? _raw_spin_lock+0x21/0x30 [ 4201.375850] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4201.379977] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4201.383949] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4201.389691] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4201.394575] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4201.403175] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4201.407032] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4201.411774] ? __next_timer_interrupt+0x160/0x160 [ 4201.415853] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4201.418038] ? do_wait_intr+0xf0/0xf0 [ 4201.420777] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4201.429769] ? kthread+0x1d1/0x200 [ 4201.431691] ? set_kthread_struct+0x70/0x70 [ 4201.433668] ? ret_from_fork+0x1f/0x30 [ 4204.060740] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 4204.074786] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4204.107658] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4204.115085] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4204.121709] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4204.124795] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4204.127420] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4204.135116] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4204.140304] RAX: 00000000000007d0 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4204.143230] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4204.146222] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4204.149493] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4204.153153] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4204.159116] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4204.162760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4204.166342] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4204.174068] Call Trace: [ 4204.178525] [ 4204.180779] ? show_regs.cold.9+0x22/0x2f [ 4204.182316] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4204.184405] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4204.186795] ? watchdog+0x40/0x40 [ 4204.188384] ? hrtimer_interrupt+0x128/0x2c0 [ 4204.190593] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4204.193377] ? apic_timer_interrupt+0xf/0x20 [ 4204.202083] [ 4204.203287] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4204.211747] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4204.216612] do_raw_spin_lock+0x177/0x190 [ 4204.220654] _raw_spin_lock+0x21/0x30 [ 4204.222007] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4204.228718] process_one_work+0x2c8/0x700 [ 4204.233571] worker_thread+0x296/0x6e0 [ 4204.236578] ? rescuer_thread+0x570/0x570 [ 4204.243262] kthread+0x1d1/0x200 [ 4204.244420] ? set_kthread_struct+0x70/0x70 [ 4204.250276] ret_from_fork+0x1f/0x30 [ 4216.062011] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4216.065941] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4216.094557] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4216.104188] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4216.118725] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4216.126734] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4216.150379] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4216.161084] RAX: 0000000000004f12 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4216.176498] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4216.185217] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4216.191688] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4216.204257] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4216.211740] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4216.216028] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4216.222586] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4216.228166] Call Trace: [ 4216.230253] [ 4216.230904] ? show_regs.cold.9+0x22/0x2f [ 4216.244099] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4216.249471] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4216.255431] ? watchdog+0x40/0x40 [ 4216.260207] ? hrtimer_interrupt+0x128/0x2c0 [ 4216.261738] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4216.264782] ? apic_timer_interrupt+0xf/0x20 [ 4216.268666] [ 4216.270710] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4216.273217] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4216.275858] do_raw_spin_lock+0x177/0x190 [ 4216.277716] _raw_spin_lock+0x21/0x30 [ 4216.279809] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4216.284192] ? do_raw_spin_unlock+0x75/0x190 [ 4216.286490] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4216.290959] ? do_wait_intr+0xf0/0xf0 [ 4216.292424] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4216.295197] mdc_reint+0x2d/0x110 [mdc] [ 4216.300949] mdc_unlink+0x658/0x870 [mdc] [ 4216.303453] lmv_unlink+0x34d/0xbd0 [lmv] [ 4216.313980] ll_unlink+0x24d/0x7f0 [lustre] [ 4216.324546] vfs_unlink+0x202/0x330 [ 4216.325944] do_unlinkat+0x2fa/0x460 [ 4216.333372] __x64_sys_unlink+0x37/0x50 [ 4216.335487] do_syscall_64+0xc1/0x440 [ 4216.337986] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4216.344077] RIP: 0033:0x7f83e5f5c10b [ 4216.349217] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4216.365406] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4216.370433] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4216.373639] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4216.377198] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4216.380084] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4216.385821] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4218.850590] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1705s! [ 4218.855506] Showing busy workqueues and worker pools: [ 4218.858016] workqueue events: flags=0x0 [ 4218.860059] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4218.862936] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4218.868161] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4218.872205] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4218.875223] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4218.886632] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4218.889944] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4218.893007] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4218.903205] workqueue events_long: flags=0x0 [ 4218.904974] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4218.909910] pending: lu_objects_flush [obdclass] [ 4218.913715] workqueue events_unbound: flags=0x2 [ 4218.914842] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4218.918229] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4218.919672] workqueue events_power_efficient: flags=0x80 [ 4218.921681] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4218.930541] pending: neigh_periodic_work, do_cache_clean [ 4218.935058] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4218.941398] pending: neigh_periodic_work, check_lifetime [ 4218.949505] workqueue mm_percpu_wq: flags=0x8 [ 4218.953755] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4218.957821] pending: vmstat_update, lru_add_drain_per_cpu [ 4218.960244] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4218.963850] pending: vmstat_update [ 4218.965384] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4218.969055] pending: lru_add_drain_per_cpu BAR(48) [ 4218.972532] workqueue ipv6_addrconf: flags=0x40008 [ 4218.974371] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4218.977199] pending: addrconf_verify_work [ 4218.979086] workqueue knbd0-recv: flags=0x1a [ 4218.981774] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4218.986419] in-flight: 650:recv_work [ 4218.988955] workqueue ptlrpc_pinger: flags=0x2 [ 4218.995286] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4218.999833] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4219.003953] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1705s workers=4 idle: 36 466 29757 [ 4219.006967] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4209s workers=2 idle: 652 [ 4228.059031] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 4228.061525] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4228.078192] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4228.081901] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4228.085386] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4228.086871] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4228.094576] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4228.097423] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4228.099932] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4228.102025] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4228.104181] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4228.106636] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4228.108879] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4228.111535] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4228.113707] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4228.116146] Call Trace: [ 4228.117314] [ 4228.118203] ? show_regs.cold.9+0x22/0x2f [ 4228.119978] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4228.121843] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4228.123538] ? watchdog+0x40/0x40 [ 4228.124997] ? hrtimer_interrupt+0x128/0x2c0 [ 4228.126546] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4228.128103] ? apic_timer_interrupt+0xf/0x20 [ 4228.129552] [ 4228.130214] ? native_safe_halt+0x2e/0x50 [ 4228.131906] kvm_wait+0xcd/0xe0 [ 4228.133157] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4228.136406] do_raw_spin_lock+0x177/0x190 [ 4228.138127] _raw_spin_lock+0x21/0x30 [ 4228.142558] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4228.146802] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4228.153146] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4228.156571] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4228.160714] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4228.163744] ? woken_wake_function+0x30/0x30 [ 4228.166660] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4228.172162] kthread+0x1d1/0x200 [ 4228.175218] ? set_kthread_struct+0x70/0x70 [ 4228.177254] ret_from_fork+0x1f/0x30 [ 4232.060207] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 4232.063312] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4232.088198] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4232.093364] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4232.097679] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4232.101674] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 4232.104046] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 4232.112840] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4232.116047] RAX: 00000000000054e5 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4232.119050] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4232.122299] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4232.125420] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4232.128483] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4232.133033] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4232.139289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4232.142884] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4232.146472] Call Trace: [ 4232.147464] [ 4232.148505] ? show_regs.cold.9+0x22/0x2f [ 4232.150752] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4232.153016] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4232.155189] ? watchdog+0x40/0x40 [ 4232.156646] ? hrtimer_interrupt+0x128/0x2c0 [ 4232.158661] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4232.161437] ? apic_timer_interrupt+0xf/0x20 [ 4232.164063] [ 4232.165085] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 4232.167599] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4232.169633] do_raw_spin_lock+0x177/0x190 [ 4232.174442] _raw_spin_lock+0x21/0x30 [ 4232.178403] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4232.182167] process_one_work+0x2c8/0x700 [ 4232.184259] worker_thread+0x296/0x6e0 [ 4232.185840] ? rescuer_thread+0x570/0x570 [ 4232.187486] kthread+0x1d1/0x200 [ 4232.188760] ? set_kthread_struct+0x70/0x70 [ 4232.190325] ret_from_fork+0x1f/0x30 [ 4244.064925] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4244.065041] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 4244.086794] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4244.086870] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4244.086878] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4244.086881] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4244.086895] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4244.086899] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4244.086904] RAX: 0000000000007087 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4244.086906] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4244.086909] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4244.086911] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4244.086913] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4244.086916] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4244.086919] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4244.086922] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4244.086931] Call Trace: [ 4244.086936] [ 4244.086942] ? show_regs.cold.9+0x22/0x2f [ 4244.086952] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4244.086959] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4244.086963] ? watchdog+0x40/0x40 [ 4244.086967] ? hrtimer_interrupt+0x128/0x2c0 [ 4244.086971] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4244.086978] ? apic_timer_interrupt+0xf/0x20 [ 4244.086995] [ 4244.086999] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4244.087005] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4244.087010] do_raw_spin_lock+0x177/0x190 [ 4244.087016] _raw_spin_lock+0x21/0x30 [ 4244.087022] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4244.087314] ? do_raw_spin_unlock+0x75/0x190 [ 4244.087321] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4244.087613] ? do_wait_intr+0xf0/0xf0 [ 4244.087623] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4244.087830] mdc_reint+0x2d/0x110 [mdc] [ 4244.087869] mdc_unlink+0x658/0x870 [mdc] [ 4244.087895] lmv_unlink+0x34d/0xbd0 [lmv] [ 4244.087926] ll_unlink+0x24d/0x7f0 [lustre] [ 4244.088044] vfs_unlink+0x202/0x330 [ 4244.088053] do_unlinkat+0x2fa/0x460 [ 4244.088058] __x64_sys_unlink+0x37/0x50 [ 4244.088061] do_syscall_64+0xc1/0x440 [ 4244.088069] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4244.088075] RIP: 0033:0x7f83e5f5c10b [ 4244.088081] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4244.088085] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4244.088090] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4244.088092] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4244.088094] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4244.088096] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4244.088099] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4244.687236] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4244.783618] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4244.800362] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4244.813247] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4244.823341] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4244.861044] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4244.872363] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4244.892928] RDX: 0000000000007c6d RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4244.905821] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4244.918700] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4244.939041] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4244.950408] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4244.962574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4244.975166] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4244.991606] Call Trace: [ 4244.998687] [ 4245.002410] ? show_regs.cold.9+0x22/0x2f [ 4245.011145] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4245.021640] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4245.031924] ? watchdog+0x40/0x40 [ 4245.037554] ? hrtimer_interrupt+0x128/0x2c0 [ 4245.042407] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4245.050698] ? apic_timer_interrupt+0xf/0x20 [ 4245.059396] [ 4245.065463] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4245.076570] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4245.092144] do_raw_spin_lock+0x177/0x190 [ 4245.103200] _raw_spin_lock+0x21/0x30 [ 4245.106288] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4245.111702] obd_mod_free+0x46/0x140 [ptlrpc] [ 4245.118665] mdc_commit_open+0x91/0xb0 [mdc] [ 4245.119724] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4245.127039] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4245.137687] after_reply+0xe2c/0x1780 [ptlrpc] [ 4245.140935] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4245.151060] ? __next_timer_interrupt+0x160/0x160 [ 4245.152967] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4245.154614] ? do_wait_intr+0xf0/0xf0 [ 4245.167990] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4245.173725] kthread+0x1d1/0x200 [ 4245.176652] ? set_kthread_struct+0x70/0x70 [ 4245.182415] ret_from_fork+0x1f/0x30 [ 4249.064717] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1735s! [ 4249.085472] Showing busy workqueues and worker pools: [ 4249.104408] workqueue events: flags=0x0 [ 4249.119812] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4249.144814] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4249.177699] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4249.204457] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4249.223337] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4249.240260] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4249.259343] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4249.272539] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4249.293688] workqueue events_long: flags=0x0 [ 4249.299927] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4249.302238] pending: lu_objects_flush [obdclass] [ 4249.316441] workqueue events_unbound: flags=0x2 [ 4249.321991] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4249.325173] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4249.331303] workqueue events_power_efficient: flags=0x80 [ 4249.333633] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4249.340661] pending: neigh_periodic_work, do_cache_clean [ 4249.346798] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4249.352568] pending: neigh_periodic_work, check_lifetime [ 4249.356395] workqueue mm_percpu_wq: flags=0x8 [ 4249.362034] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4249.370535] pending: vmstat_update, lru_add_drain_per_cpu [ 4249.377822] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4249.383592] pending: vmstat_update [ 4249.386780] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4249.397331] pending: lru_add_drain_per_cpu BAR(48) [ 4249.408289] workqueue ipv6_addrconf: flags=0x40008 [ 4249.413335] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4249.422607] pending: addrconf_verify_work [ 4249.424691] workqueue knbd0-recv: flags=0x1a [ 4249.430682] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4249.434239] in-flight: 650:recv_work [ 4249.437487] workqueue ptlrpc_pinger: flags=0x2 [ 4249.446484] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4249.458176] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4249.464725] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1736s workers=4 idle: 36 466 29757 [ 4249.472378] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4239s workers=2 idle: 652 [ 4256.059057] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 4256.064797] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4256.093186] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4256.099468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4256.103067] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4256.104549] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4256.114388] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4256.118084] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4256.121655] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4256.125833] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4256.129184] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4256.132651] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4256.139477] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4256.147048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4256.153189] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4256.160209] Call Trace: [ 4256.161347] [ 4256.164214] ? show_regs.cold.9+0x22/0x2f [ 4256.168288] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4256.174481] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4256.178652] ? watchdog+0x40/0x40 [ 4256.181924] ? hrtimer_interrupt+0x128/0x2c0 [ 4256.187652] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4256.190498] ? apic_timer_interrupt+0xf/0x20 [ 4256.192648] [ 4256.193697] ? native_safe_halt+0x2e/0x50 [ 4256.196576] kvm_wait+0xcd/0xe0 [ 4256.198424] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4256.201217] do_raw_spin_lock+0x177/0x190 [ 4256.203317] _raw_spin_lock+0x21/0x30 [ 4256.204757] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4256.207743] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4256.210547] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4256.212941] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4256.215535] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4256.218353] ? woken_wake_function+0x30/0x30 [ 4256.219844] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4256.223100] kthread+0x1d1/0x200 [ 4256.224523] ? set_kthread_struct+0x70/0x70 [ 4256.226557] ret_from_fork+0x1f/0x30 [ 4260.060895] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 4260.070556] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4260.095388] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4260.101491] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4260.106350] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4260.108837] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4260.111595] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4260.119167] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4260.123171] RAX: 0000000000003b0e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4260.126263] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4260.129212] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4260.132282] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4260.135215] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4260.138161] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4260.141412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4260.143744] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4260.146845] Call Trace: [ 4260.147747] [ 4260.148344] ? show_regs.cold.9+0x22/0x2f [ 4260.150201] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4260.152203] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4260.153714] ? watchdog+0x40/0x40 [ 4260.154780] ? hrtimer_interrupt+0x128/0x2c0 [ 4260.157634] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4260.160587] ? apic_timer_interrupt+0xf/0x20 [ 4260.162268] [ 4260.163106] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4260.165975] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4260.168180] do_raw_spin_lock+0x177/0x190 [ 4260.169558] _raw_spin_lock+0x21/0x30 [ 4260.171121] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4260.174205] process_one_work+0x2c8/0x700 [ 4260.176442] worker_thread+0x296/0x6e0 [ 4260.178114] ? rescuer_thread+0x570/0x570 [ 4260.179740] kthread+0x1d1/0x200 [ 4260.181235] ? set_kthread_struct+0x70/0x70 [ 4260.183131] ret_from_fork+0x1f/0x30 [ 4272.065580] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4272.079063] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4272.166995] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4272.189236] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4272.201980] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4272.214781] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4272.228431] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4272.239550] RAX: 0000000000007d0f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4272.247394] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4272.250520] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4272.258564] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4272.269956] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4272.285574] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4272.295126] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4272.301979] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4272.305939] Call Trace: [ 4272.307401] [ 4272.309348] ? show_regs.cold.9+0x22/0x2f [ 4272.311382] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4272.314866] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4272.317229] ? watchdog+0x40/0x40 [ 4272.319844] ? hrtimer_interrupt+0x128/0x2c0 [ 4272.321352] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4272.325021] ? apic_timer_interrupt+0xf/0x20 [ 4272.327428] [ 4272.329322] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4272.332660] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4272.336269] do_raw_spin_lock+0x177/0x190 [ 4272.339269] _raw_spin_lock+0x21/0x30 [ 4272.341185] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4272.344680] ? do_raw_spin_unlock+0x75/0x190 [ 4272.347421] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4272.350008] ? do_wait_intr+0xf0/0xf0 [ 4272.352026] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4272.355020] mdc_reint+0x2d/0x110 [mdc] [ 4272.357377] mdc_unlink+0x658/0x870 [mdc] [ 4272.363427] lmv_unlink+0x34d/0xbd0 [lmv] [ 4272.382067] ll_unlink+0x24d/0x7f0 [lustre] [ 4272.383452] vfs_unlink+0x202/0x330 [ 4272.385052] do_unlinkat+0x2fa/0x460 [ 4272.386958] __x64_sys_unlink+0x37/0x50 [ 4272.388886] do_syscall_64+0xc1/0x440 [ 4272.391238] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4272.393984] RIP: 0033:0x7f83e5f5c10b [ 4272.395594] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4272.403860] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4272.407020] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4272.410266] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4272.414186] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4272.417988] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4272.421995] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4276.056025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 4276.058737] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4276.078666] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4276.083697] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4276.087314] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4276.089245] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4276.096935] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4276.099326] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4276.102382] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4276.105257] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4276.107386] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4276.111285] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4276.113676] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4276.117264] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4276.119060] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4276.121159] Call Trace: [ 4276.122755] [ 4276.123403] ? show_regs.cold.9+0x22/0x2f [ 4276.124728] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4276.126329] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4276.128103] ? watchdog+0x40/0x40 [ 4276.129106] ? hrtimer_interrupt+0x128/0x2c0 [ 4276.130629] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4276.132818] ? apic_timer_interrupt+0xf/0x20 [ 4276.134345] [ 4276.135826] ? native_safe_halt+0x2e/0x50 [ 4276.137483] kvm_wait+0xcd/0xe0 [ 4276.138961] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4276.141372] do_raw_spin_lock+0x177/0x190 [ 4276.142914] _raw_spin_lock+0x21/0x30 [ 4276.144190] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4276.147659] obd_mod_free+0x46/0x140 [ptlrpc] [ 4276.151323] mdc_commit_open+0x91/0xb0 [mdc] [ 4276.152838] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4276.155397] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4276.157833] after_reply+0xe2c/0x1780 [ptlrpc] [ 4276.160351] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4276.162242] ? __next_timer_interrupt+0x160/0x160 [ 4276.165406] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4276.167130] ? do_wait_intr+0xf0/0xf0 [ 4276.168886] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4276.170678] kthread+0x1d1/0x200 [ 4276.172778] ? set_kthread_struct+0x70/0x70 [ 4276.174177] ret_from_fork+0x1f/0x30 [ 4279.784622] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1766s! [ 4279.791308] Showing busy workqueues and worker pools: [ 4279.803465] workqueue events: flags=0x0 [ 4279.806031] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4279.816845] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4279.826337] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4279.830701] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4279.836248] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4279.839157] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4279.847633] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4279.855423] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4279.865319] workqueue events_long: flags=0x0 [ 4279.868561] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4279.874721] pending: lu_objects_flush [obdclass] [ 4279.882647] workqueue events_unbound: flags=0x2 [ 4279.884307] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4279.891375] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4279.893034] workqueue events_power_efficient: flags=0x80 [ 4279.904838] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4279.906585] pending: neigh_periodic_work, do_cache_clean [ 4279.908531] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4279.910503] pending: neigh_periodic_work, check_lifetime [ 4279.917011] workqueue mm_percpu_wq: flags=0x8 [ 4279.919457] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4279.925387] pending: vmstat_update, lru_add_drain_per_cpu [ 4279.930165] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4279.933724] pending: vmstat_update [ 4279.936960] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4279.943760] pending: lru_add_drain_per_cpu BAR(48) [ 4279.949126] workqueue ipv6_addrconf: flags=0x40008 [ 4279.957254] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4279.966104] pending: addrconf_verify_work [ 4279.968993] workqueue knbd0-recv: flags=0x1a [ 4279.974888] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4279.979271] in-flight: 650:recv_work [ 4279.980406] workqueue ptlrpc_pinger: flags=0x2 [ 4279.985338] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4279.992371] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4279.995781] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1766s workers=4 idle: 36 466 29757 [ 4280.004833] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4270s workers=2 idle: 652 [ 4284.062617] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 4284.071791] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4284.105979] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4284.113008] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4284.117229] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4284.119580] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4284.133474] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4284.138078] RAX: 0000000000004132 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4284.145004] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4284.148913] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4284.152879] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4284.158089] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4284.162548] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4284.169556] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4284.173840] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4284.181151] Call Trace: [ 4284.182970] [ 4284.184334] ? show_regs.cold.9+0x22/0x2f [ 4284.187290] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4284.190276] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4284.197178] ? watchdog+0x40/0x40 [ 4284.198838] ? hrtimer_interrupt+0x128/0x2c0 [ 4284.202006] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4284.205099] ? apic_timer_interrupt+0xf/0x20 [ 4284.207860] [ 4284.209472] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4284.212890] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4284.216530] do_raw_spin_lock+0x177/0x190 [ 4284.219210] _raw_spin_lock+0x21/0x30 [ 4284.221208] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4284.226175] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4284.229788] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4284.233290] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4284.237999] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4284.241225] ? woken_wake_function+0x30/0x30 [ 4284.246580] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4284.261471] kthread+0x1d1/0x200 [ 4284.270991] ? set_kthread_struct+0x70/0x70 [ 4284.277018] ret_from_fork+0x1f/0x30 [ 4288.060150] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 4288.064466] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4288.110460] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4288.123424] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4288.143091] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4288.151352] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4288.156629] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4288.171344] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4288.177797] RAX: 000000000000541e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4288.185411] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4288.199205] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4288.203368] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4288.216698] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4288.219029] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4288.227379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4288.234299] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4288.244251] Call Trace: [ 4288.246602] [ 4288.249434] ? show_regs.cold.9+0x22/0x2f [ 4288.252008] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4288.260792] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4288.266526] ? watchdog+0x40/0x40 [ 4288.271414] ? hrtimer_interrupt+0x128/0x2c0 [ 4288.275895] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4288.279732] ? apic_timer_interrupt+0xf/0x20 [ 4288.282866] [ 4288.283541] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4288.289784] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4288.296056] do_raw_spin_lock+0x177/0x190 [ 4288.305453] _raw_spin_lock+0x21/0x30 [ 4288.309025] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4288.324466] process_one_work+0x2c8/0x700 [ 4288.330945] worker_thread+0x296/0x6e0 [ 4288.334509] ? rescuer_thread+0x570/0x570 [ 4288.341224] kthread+0x1d1/0x200 [ 4288.345723] ? set_kthread_struct+0x70/0x70 [ 4288.347230] ret_from_fork+0x1f/0x30 [ 4300.062339] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4300.069094] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4300.101308] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4300.108053] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4300.113083] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4300.115500] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4300.124882] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4300.128504] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4300.132178] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4300.136380] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4300.140110] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4300.143801] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4300.147299] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4300.152745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4300.155762] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4300.159746] Call Trace: [ 4300.160721] [ 4300.161381] ? show_regs.cold.9+0x22/0x2f [ 4300.163377] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4300.165707] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4300.167822] ? watchdog+0x40/0x40 [ 4300.169548] ? hrtimer_interrupt+0x128/0x2c0 [ 4300.171807] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4300.174471] ? apic_timer_interrupt+0xf/0x20 [ 4300.176789] [ 4300.177879] ? native_safe_halt+0x2e/0x50 [ 4300.180140] kvm_wait+0xcd/0xe0 [ 4300.181845] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4300.184352] do_raw_spin_lock+0x177/0x190 [ 4300.186771] _raw_spin_lock+0x21/0x30 [ 4300.188800] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4300.192137] ? do_raw_spin_unlock+0x75/0x190 [ 4300.194803] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4300.197754] ? do_wait_intr+0xf0/0xf0 [ 4300.201074] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4300.205019] mdc_reint+0x2d/0x110 [mdc] [ 4300.208000] mdc_unlink+0x658/0x870 [mdc] [ 4300.210805] lmv_unlink+0x34d/0xbd0 [lmv] [ 4300.213769] ll_unlink+0x24d/0x7f0 [lustre] [ 4300.216973] vfs_unlink+0x202/0x330 [ 4300.219425] do_unlinkat+0x2fa/0x460 [ 4300.222786] __x64_sys_unlink+0x37/0x50 [ 4300.224965] do_syscall_64+0xc1/0x440 [ 4300.227945] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4300.231392] RIP: 0033:0x7f83e5f5c10b [ 4300.233989] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4300.248016] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4300.258050] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4300.263339] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4300.270938] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4300.277475] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4300.282473] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4304.056373] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 4304.067368] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4304.132331] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4304.151245] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4304.160135] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4304.167185] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4304.186928] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4304.195040] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4304.204067] RDX: 00000000000034cb RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4304.210590] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4304.217017] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4304.222012] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4304.225905] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4304.234986] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4304.246185] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4304.253634] Call Trace: [ 4304.256068] [ 4304.262327] ? show_regs.cold.9+0x22/0x2f [ 4304.263822] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4304.276828] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4304.284793] ? watchdog+0x40/0x40 [ 4304.292088] ? hrtimer_interrupt+0x128/0x2c0 [ 4304.303499] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4304.306837] ? apic_timer_interrupt+0xf/0x20 [ 4304.311984] [ 4304.314127] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4304.318982] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4304.327468] do_raw_spin_lock+0x177/0x190 [ 4304.333379] _raw_spin_lock+0x21/0x30 [ 4304.337978] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4304.342093] obd_mod_free+0x46/0x140 [ptlrpc] [ 4304.345822] mdc_commit_open+0x91/0xb0 [mdc] [ 4304.353542] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4304.358974] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4304.366504] after_reply+0xe2c/0x1780 [ptlrpc] [ 4304.372274] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4304.385099] ? __next_timer_interrupt+0x160/0x160 [ 4304.395596] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4304.399093] ? do_wait_intr+0xf0/0xf0 [ 4304.403181] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4304.412153] kthread+0x1d1/0x200 [ 4304.418441] ? set_kthread_struct+0x70/0x70 [ 4304.425519] ret_from_fork+0x1f/0x30 [ 4310.498385] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1797s! [ 4310.507990] Showing busy workqueues and worker pools: [ 4310.513731] workqueue events: flags=0x0 [ 4310.518708] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4310.529522] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4310.539102] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4310.548869] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4310.555090] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4310.558459] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4310.566695] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4310.572796] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4310.580120] workqueue events_long: flags=0x0 [ 4310.581602] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4310.585882] pending: lu_objects_flush [obdclass] [ 4310.588946] workqueue events_unbound: flags=0x2 [ 4310.590458] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4310.594465] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4310.597708] workqueue events_power_efficient: flags=0x80 [ 4310.601010] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4310.606131] pending: neigh_periodic_work, do_cache_clean [ 4310.612001] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4310.616175] pending: neigh_periodic_work, check_lifetime [ 4310.618885] workqueue mm_percpu_wq: flags=0x8 [ 4310.622595] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4310.625408] pending: vmstat_update, lru_add_drain_per_cpu [ 4310.630702] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4310.633548] pending: vmstat_update [ 4310.635633] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4310.642022] pending: lru_add_drain_per_cpu BAR(48) [ 4310.645592] workqueue ipv6_addrconf: flags=0x40008 [ 4310.652303] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4310.659913] pending: addrconf_verify_work [ 4310.661432] workqueue knbd0-recv: flags=0x1a [ 4310.671013] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4310.682907] in-flight: 650:recv_work [ 4310.686590] workqueue ptlrpc_pinger: flags=0x2 [ 4310.688039] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4310.693956] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4310.700098] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1797s workers=4 idle: 36 466 29757 [ 4310.705039] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4300s workers=2 idle: 652 [ 4312.059073] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 4312.062373] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4312.084722] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4312.091213] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4312.095511] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4312.098766] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4312.108494] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4312.112693] RAX: 0000000000000b9f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4312.116403] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4312.119589] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4312.122727] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4312.125677] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4312.128963] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4312.132265] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4312.134674] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4312.138222] Call Trace: [ 4312.139397] [ 4312.140272] ? show_regs.cold.9+0x22/0x2f [ 4312.141973] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4312.144282] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4312.146434] ? watchdog+0x40/0x40 [ 4312.147657] ? hrtimer_interrupt+0x128/0x2c0 [ 4312.149526] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4312.151685] ? apic_timer_interrupt+0xf/0x20 [ 4312.153676] [ 4312.154749] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4312.157261] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4312.159871] do_raw_spin_lock+0x177/0x190 [ 4312.161709] _raw_spin_lock+0x21/0x30 [ 4312.163535] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4312.166779] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4312.169439] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4312.172155] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4312.174699] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4312.177449] ? woken_wake_function+0x30/0x30 [ 4312.179253] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4312.182435] kthread+0x1d1/0x200 [ 4312.184042] ? set_kthread_struct+0x70/0x70 [ 4312.185904] ret_from_fork+0x1f/0x30 [ 4316.060107] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 4316.064449] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4316.101709] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4316.110812] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4316.115374] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4316.118526] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4316.120305] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4316.127112] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4316.130026] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4316.133709] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4316.136769] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4316.139874] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4316.142945] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4316.146150] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4316.148801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4316.150824] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4316.153014] Call Trace: [ 4316.153745] [ 4316.154496] ? show_regs.cold.9+0x22/0x2f [ 4316.155965] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4316.157847] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4316.159499] ? watchdog+0x40/0x40 [ 4316.160604] ? hrtimer_interrupt+0x128/0x2c0 [ 4316.161886] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4316.163743] ? apic_timer_interrupt+0xf/0x20 [ 4316.165396] [ 4316.166229] ? native_safe_halt+0x2e/0x50 [ 4316.167789] kvm_wait+0xcd/0xe0 [ 4316.168783] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4316.170761] do_raw_spin_lock+0x177/0x190 [ 4316.172263] _raw_spin_lock+0x21/0x30 [ 4316.173820] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4316.175738] process_one_work+0x2c8/0x700 [ 4316.177097] worker_thread+0x296/0x6e0 [ 4316.178374] ? rescuer_thread+0x570/0x570 [ 4316.179641] kthread+0x1d1/0x200 [ 4316.180861] ? set_kthread_struct+0x70/0x70 [ 4316.182364] ret_from_fork+0x1f/0x30 [ 4328.062104] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4328.065800] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4328.081541] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4328.084874] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4328.089094] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4328.090929] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4328.098580] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4328.102063] RAX: 0000000000003ef1 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4328.107588] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4328.112611] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4328.117567] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4328.120176] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4328.123195] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4328.127938] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4328.130951] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4328.135295] Call Trace: [ 4328.137260] [ 4328.138426] ? show_regs.cold.9+0x22/0x2f [ 4328.140777] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4328.143333] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4328.145865] ? watchdog+0x40/0x40 [ 4328.147782] ? hrtimer_interrupt+0x128/0x2c0 [ 4328.149832] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4328.153259] ? apic_timer_interrupt+0xf/0x20 [ 4328.155755] [ 4328.156949] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4328.160128] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4328.162593] do_raw_spin_lock+0x177/0x190 [ 4328.164918] _raw_spin_lock+0x21/0x30 [ 4328.166955] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4328.170097] ? do_raw_spin_unlock+0x75/0x190 [ 4328.172448] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4328.175132] ? do_wait_intr+0xf0/0xf0 [ 4328.177228] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4328.179939] mdc_reint+0x2d/0x110 [mdc] [ 4328.181395] mdc_unlink+0x658/0x870 [mdc] [ 4328.182811] lmv_unlink+0x34d/0xbd0 [lmv] [ 4328.185045] ll_unlink+0x24d/0x7f0 [lustre] [ 4328.187720] vfs_unlink+0x202/0x330 [ 4328.190313] do_unlinkat+0x2fa/0x460 [ 4328.192779] __x64_sys_unlink+0x37/0x50 [ 4328.195823] do_syscall_64+0xc1/0x440 [ 4328.197707] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4328.200601] RIP: 0033:0x7f83e5f5c10b [ 4328.202919] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4328.217719] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4328.221837] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4328.226103] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4328.228809] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4328.232276] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4328.236208] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4336.056029] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 4336.059366] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4336.082682] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4336.088242] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4336.093036] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4336.095388] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4336.105352] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4336.109400] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4336.112680] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4336.114765] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4336.117459] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4336.120915] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4336.124395] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4336.128373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4336.131386] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4336.135688] Call Trace: [ 4336.136902] [ 4336.138931] ? show_regs.cold.9+0x22/0x2f [ 4336.141635] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4336.146534] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4336.149295] ? watchdog+0x40/0x40 [ 4336.152727] ? hrtimer_interrupt+0x128/0x2c0 [ 4336.154748] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4336.157386] ? apic_timer_interrupt+0xf/0x20 [ 4336.159271] [ 4336.160819] ? native_safe_halt+0x2e/0x50 [ 4336.163080] kvm_wait+0xcd/0xe0 [ 4336.164524] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4336.167053] do_raw_spin_lock+0x177/0x190 [ 4336.169205] _raw_spin_lock+0x21/0x30 [ 4336.171048] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4336.173404] obd_mod_free+0x46/0x140 [ptlrpc] [ 4336.176027] mdc_commit_open+0x91/0xb0 [mdc] [ 4336.178105] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4336.180916] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4336.184293] after_reply+0xe2c/0x1780 [ptlrpc] [ 4336.186821] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4336.189704] ? __next_timer_interrupt+0x160/0x160 [ 4336.192187] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4336.194449] ? do_wait_intr+0xf0/0xf0 [ 4336.196280] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4336.198680] kthread+0x1d1/0x200 [ 4336.200611] ? set_kthread_struct+0x70/0x70 [ 4336.202559] ret_from_fork+0x1f/0x30 [ 4340.059289] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 4340.062802] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4340.084179] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4340.088624] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4340.097412] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4340.101241] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4340.110184] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4340.114131] RAX: 000000000000428c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4340.118803] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4340.122496] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4340.126380] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4340.135404] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4340.142183] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4340.149656] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4340.153031] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4340.156207] Call Trace: [ 4340.157033] [ 4340.161108] ? show_regs.cold.9+0x22/0x2f [ 4340.163395] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4340.170735] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4340.176181] ? watchdog+0x40/0x40 [ 4340.178355] ? hrtimer_interrupt+0x128/0x2c0 [ 4340.181114] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4340.185178] ? apic_timer_interrupt+0xf/0x20 [ 4340.187835] [ 4340.189461] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4340.190805] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4340.192729] do_raw_spin_lock+0x177/0x190 [ 4340.195369] _raw_spin_lock+0x21/0x30 [ 4340.197248] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4340.200622] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4340.203386] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4340.205802] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4340.209007] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4340.210977] ? woken_wake_function+0x30/0x30 [ 4340.212416] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4340.214905] kthread+0x1d1/0x200 [ 4340.216544] ? set_kthread_struct+0x70/0x70 [ 4340.218327] ret_from_fork+0x1f/0x30 [ 4341.217074] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1828s! [ 4341.220323] Showing busy workqueues and worker pools: [ 4341.222390] workqueue events: flags=0x0 [ 4341.223908] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4341.226666] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4341.230439] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4341.233155] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4341.237132] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4341.241220] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4341.247830] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4341.252181] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4341.257263] workqueue events_long: flags=0x0 [ 4341.259350] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4341.264527] pending: lu_objects_flush [obdclass] [ 4341.267515] workqueue events_unbound: flags=0x2 [ 4341.270104] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4341.274003] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4341.277598] workqueue events_power_efficient: flags=0x80 [ 4341.280278] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4341.284591] pending: neigh_periodic_work, do_cache_clean [ 4341.291125] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4341.296341] pending: neigh_periodic_work, check_lifetime [ 4341.300330] workqueue mm_percpu_wq: flags=0x8 [ 4341.303649] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4341.308407] pending: vmstat_update, lru_add_drain_per_cpu [ 4341.314339] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4341.318516] pending: vmstat_update [ 4341.320262] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4341.323964] pending: lru_add_drain_per_cpu BAR(48) [ 4341.327342] workqueue ipv6_addrconf: flags=0x40008 [ 4341.330325] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4341.334474] pending: addrconf_verify_work [ 4341.337237] workqueue knbd0-recv: flags=0x1a [ 4341.339813] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4341.344184] in-flight: 650:recv_work [ 4341.347448] workqueue ptlrpc_pinger: flags=0x2 [ 4341.349522] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4341.354993] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4341.359386] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1828s workers=4 idle: 36 466 29757 [ 4341.362370] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4331s workers=2 idle: 652 [ 4344.060135] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 4344.066781] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4344.111780] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4344.121421] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4344.130377] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4344.135154] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4344.138760] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4344.160953] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4344.166188] RAX: 000000000000564b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4344.170691] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4344.180718] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4344.189986] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4344.195026] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4344.199219] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4344.216410] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4344.225096] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4344.227424] Call Trace: [ 4344.233036] [ 4344.234802] ? show_regs.cold.9+0x22/0x2f [ 4344.242707] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4344.244369] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4344.252680] ? watchdog+0x40/0x40 [ 4344.255768] ? hrtimer_interrupt+0x128/0x2c0 [ 4344.259298] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4344.262740] ? apic_timer_interrupt+0xf/0x20 [ 4344.269110] [ 4344.271539] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4344.278879] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4344.285134] do_raw_spin_lock+0x177/0x190 [ 4344.288748] _raw_spin_lock+0x21/0x30 [ 4344.291369] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4344.301921] process_one_work+0x2c8/0x700 [ 4344.304826] worker_thread+0x296/0x6e0 [ 4344.307835] ? rescuer_thread+0x570/0x570 [ 4344.315087] kthread+0x1d1/0x200 [ 4344.317919] ? set_kthread_struct+0x70/0x70 [ 4344.321823] ret_from_fork+0x1f/0x30 [ 4356.062025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4356.064183] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4356.108440] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4356.115576] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4356.120423] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4356.123677] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4356.142923] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4356.145451] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4356.158480] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4356.163208] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4356.168352] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4356.174081] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4356.176853] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4356.195054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4356.196528] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4356.200526] Call Trace: [ 4356.208385] [ 4356.208934] ? show_regs.cold.9+0x22/0x2f [ 4356.209717] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4356.212499] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4356.216267] ? watchdog+0x40/0x40 [ 4356.218135] ? hrtimer_interrupt+0x128/0x2c0 [ 4356.220260] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4356.222744] ? apic_timer_interrupt+0xf/0x20 [ 4356.224749] [ 4356.225560] ? native_safe_halt+0x2e/0x50 [ 4356.227377] kvm_wait+0xcd/0xe0 [ 4356.229095] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4356.231187] do_raw_spin_lock+0x177/0x190 [ 4356.232410] _raw_spin_lock+0x21/0x30 [ 4356.233789] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4356.236780] ? do_raw_spin_unlock+0x75/0x190 [ 4356.238760] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4356.241802] ? do_wait_intr+0xf0/0xf0 [ 4356.244586] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4356.249186] mdc_reint+0x2d/0x110 [mdc] [ 4356.250426] mdc_unlink+0x658/0x870 [mdc] [ 4356.253722] lmv_unlink+0x34d/0xbd0 [lmv] [ 4356.255056] ll_unlink+0x24d/0x7f0 [lustre] [ 4356.258296] vfs_unlink+0x202/0x330 [ 4356.259293] do_unlinkat+0x2fa/0x460 [ 4356.261254] __x64_sys_unlink+0x37/0x50 [ 4356.262248] do_syscall_64+0xc1/0x440 [ 4356.263192] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4356.264401] RIP: 0033:0x7f83e5f5c10b [ 4356.265435] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4356.279222] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4356.286077] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4356.294079] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4356.299049] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4356.302498] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4356.307066] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4368.057052] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 4368.059061] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 4368.067165] Modules linked in: lustre(O) [ 4368.074397] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4368.074478] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4368.074486] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4368.074489] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4368.074502] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4368.074507] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4368.074512] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4368.074514] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4368.074516] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4368.074518] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4368.074520] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4368.074523] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4368.074525] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4368.074528] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4368.074537] Call Trace: [ 4368.074542] [ 4368.074547] ? show_regs.cold.9+0x22/0x2f [ 4368.074557] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4368.074563] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4368.074567] ? watchdog+0x40/0x40 [ 4368.074571] ? hrtimer_interrupt+0x128/0x2c0 [ 4368.074576] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4368.074596] ? apic_timer_interrupt+0xf/0x20 [ 4368.074600] [ 4368.074603] ? native_safe_halt+0x2e/0x50 [ 4368.074608] kvm_wait+0xcd/0xe0 [ 4368.074615] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4368.074624] do_raw_spin_lock+0x177/0x190 [ 4368.074628] _raw_spin_lock+0x21/0x30 [ 4368.074632] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4368.074900] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4368.075090] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4368.077924] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4368.077995] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4368.078101] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4368.078105] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4368.078118] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4368.078122] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4368.078126] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4368.078129] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4368.078131] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4368.078133] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4368.078136] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4368.078139] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4368.078142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4368.078144] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4368.078152] Call Trace: [ 4368.078157] [ 4368.078163] ? show_regs.cold.9+0x22/0x2f [ 4368.078173] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4368.078179] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4368.078184] ? watchdog+0x40/0x40 [ 4368.078188] ? hrtimer_interrupt+0x128/0x2c0 [ 4368.078192] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4368.078198] ? apic_timer_interrupt+0xf/0x20 [ 4368.078203] [ 4368.078206] ? native_safe_halt+0x2e/0x50 [ 4368.078209] kvm_wait+0xcd/0xe0 [ 4368.078216] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4368.078224] do_raw_spin_lock+0x177/0x190 [ 4368.078228] _raw_spin_lock+0x21/0x30 [ 4368.078233] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4368.078507] obd_mod_free+0x46/0x140 [ptlrpc] [ 4368.078941] mdc_commit_open+0x91/0xb0 [mdc] [ 4368.079071] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4368.079264] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4368.079515] after_reply+0xe2c/0x1780 [ptlrpc] [ 4368.079883] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4368.080164] ? __next_timer_interrupt+0x160/0x160 [ 4368.080174] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4368.099861] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4368.108285] ? do_wait_intr+0xf0/0xf0 [ 4368.110657] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4368.111138] ? woken_wake_function+0x30/0x30 [ 4368.111147] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4368.111362] kthread+0x1d1/0x200 [ 4368.111369] ? set_kthread_struct+0x70/0x70 [ 4368.111373] ret_from_fork+0x1f/0x30 [ 4368.453638] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4368.457399] kthread+0x1d1/0x200 [ 4368.459377] ? set_kthread_struct+0x70/0x70 [ 4368.465529] ret_from_fork+0x1f/0x30 [ 4371.429026] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1858s! [ 4371.433163] Showing busy workqueues and worker pools: [ 4371.434959] workqueue events: flags=0x0 [ 4371.436304] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4371.440411] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4371.448531] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4371.453387] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4371.456662] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4371.460700] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4371.464818] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4371.469649] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4371.474534] workqueue events_long: flags=0x0 [ 4371.476976] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4371.481535] pending: lu_objects_flush [obdclass] [ 4371.484805] workqueue events_unbound: flags=0x2 [ 4371.487530] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4371.490902] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4371.493396] workqueue events_power_efficient: flags=0x80 [ 4371.496983] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4371.499900] pending: neigh_periodic_work, do_cache_clean [ 4371.502174] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4371.505520] pending: neigh_periodic_work, check_lifetime [ 4371.508246] workqueue mm_percpu_wq: flags=0x8 [ 4371.510446] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4371.514275] pending: vmstat_update, lru_add_drain_per_cpu [ 4371.517692] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4371.521115] pending: vmstat_update [ 4371.522920] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4371.526342] pending: lru_add_drain_per_cpu BAR(48) [ 4371.529062] workqueue ipv6_addrconf: flags=0x40008 [ 4371.531845] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4371.535536] pending: addrconf_verify_work [ 4371.538162] workqueue knbd0-recv: flags=0x1a [ 4371.543843] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4371.546266] in-flight: 650:recv_work [ 4371.548057] workqueue ptlrpc_pinger: flags=0x2 [ 4371.552483] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4371.558976] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4371.566872] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1858s workers=4 idle: 36 466 29757 [ 4371.574013] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4361s workers=2 idle: 652 [ 4372.060790] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 4372.079769] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4372.133273] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4372.144320] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4372.149958] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4372.156511] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4372.160241] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4372.176722] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4372.179501] RAX: 0000000000004288 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4372.185006] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4372.189329] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4372.193395] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4372.198302] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4372.200149] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4372.208399] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4372.210484] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4372.219927] Call Trace: [ 4372.221351] [ 4372.222150] ? show_regs.cold.9+0x22/0x2f [ 4372.226997] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4372.234001] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4372.237084] ? watchdog+0x40/0x40 [ 4372.238455] ? hrtimer_interrupt+0x128/0x2c0 [ 4372.242121] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4372.244418] ? apic_timer_interrupt+0xf/0x20 [ 4372.247071] [ 4372.249654] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4372.253503] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4372.255370] do_raw_spin_lock+0x177/0x190 [ 4372.259829] _raw_spin_lock+0x21/0x30 [ 4372.260955] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4372.274635] process_one_work+0x2c8/0x700 [ 4372.276091] worker_thread+0x296/0x6e0 [ 4372.277427] ? rescuer_thread+0x570/0x570 [ 4372.283324] kthread+0x1d1/0x200 [ 4372.285124] ? set_kthread_struct+0x70/0x70 [ 4372.286987] ret_from_fork+0x1f/0x30 [ 4381.010917] rcu: INFO: rcu_sched self-detected stall on CPU [ 4381.022930] rcu: 0-....: (1498226 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 4381.037370] (t=2400078 jiffies g=262441 q=57761) [ 4381.039228] rcu: rcu_sched kthread starved for 1869267 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 4381.043922] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4381.047047] rcu: RCU grace-period kthread stack dump: [ 4381.048988] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4381.052137] Call Trace: [ 4381.053018] __schedule+0x351/0xcb0 [ 4381.055003] schedule+0xc0/0x180 [ 4381.056128] schedule_timeout+0xb4/0x190 [ 4381.057585] ? do_raw_spin_unlock+0x75/0x190 [ 4381.059040] ? __next_timer_interrupt+0x160/0x160 [ 4381.060948] ? prepare_to_swait_event+0xf3/0x140 [ 4381.062870] rcu_gp_kthread+0x8dd/0x15b0 [ 4381.064365] ? force_qs_rnp+0x2c0/0x2c0 [ 4381.068935] kthread+0x1d1/0x200 [ 4381.073102] ? set_kthread_struct+0x70/0x70 [ 4381.078844] ret_from_fork+0x1f/0x30 [ 4381.082856] rcu: Stack dump where RCU GP kthread last ran: [ 4381.092167] Sending NMI from CPU 0 to CPUs 1: [ 4381.102775] NMI backtrace for cpu 1 [ 4381.102782] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4381.102784] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4381.102786] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4381.102787] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4381.102789] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 [ 4381.102791] RAX: 0000000000004685 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4381.102793] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4381.102794] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4381.102795] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4381.102796] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4381.102797] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4381.102799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4381.102800] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4381.102801] Call Trace: [ 4381.102802] [ 4381.102803] ? show_regs.cold.9+0x22/0x2f [ 4381.102804] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4381.102805] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4381.102806] ? nmi_handle+0xbd/0x210 [ 4381.102807] ? default_do_nmi+0x4b/0x1b0 [ 4381.102808] ? do_nmi+0x280/0x380 [ 4381.102809] ? end_repeat_nmi+0x16/0x1f [ 4381.102809] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4381.102810] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4381.102811] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4381.102812] [ 4381.102813] do_raw_spin_lock+0x177/0x190 [ 4381.102814] _raw_spin_lock+0x21/0x30 [ 4381.102815] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4381.102816] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4381.102817] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4381.102818] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4381.102818] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4381.102819] ? woken_wake_function+0x30/0x30 [ 4381.102820] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4381.102821] kthread+0x1d1/0x200 [ 4381.102822] ? set_kthread_struct+0x70/0x70 [ 4381.102823] ret_from_fork+0x1f/0x30 [ 4381.108852] NMI backtrace for cpu 0 [ 4381.277298] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4381.284026] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4381.286939] Call Trace: [ 4381.288899] [ 4381.289946] ? dump_stack+0xbb/0x10e [ 4381.291636] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4381.292949] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4381.295095] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4381.301231] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4381.307126] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4381.310139] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4381.313186] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4381.315934] ? update_process_times+0x6d/0xc0 [ 4381.318306] ? tick_sched_handle.isra.18+0x27/0x90 [ 4381.321234] ? tick_sched_timer+0x53/0xb0 [ 4381.323957] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4381.326202] ? tick_sched_handle.isra.18+0x90/0x90 [ 4381.329325] ? hrtimer_interrupt+0x128/0x2c0 [ 4381.331979] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4381.334275] ? apic_timer_interrupt+0xf/0x20 [ 4381.336244] [ 4381.337083] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4381.339714] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4381.341927] ? do_raw_spin_lock+0x177/0x190 [ 4381.343646] ? _raw_spin_lock+0x21/0x30 [ 4381.345424] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4381.348108] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4381.350164] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4381.352328] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4381.355271] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4381.358112] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4381.360639] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4381.363879] ? __next_timer_interrupt+0x160/0x160 [ 4381.366072] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4381.368685] ? do_wait_intr+0xf0/0xf0 [ 4381.370722] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4381.373510] ? kthread+0x1d1/0x200 [ 4381.377673] ? set_kthread_struct+0x70/0x70 [ 4381.380496] ? ret_from_fork+0x1f/0x30 [ 4384.064999] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4384.070174] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4384.096724] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4384.100012] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4384.105342] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4384.109966] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4384.119313] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4384.128145] RAX: 0000000000006d24 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4384.130677] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4384.133247] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4384.137034] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4384.141574] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4384.146824] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4384.149331] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4384.155075] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4384.157227] Call Trace: [ 4384.157906] [ 4384.158746] ? show_regs.cold.9+0x22/0x2f [ 4384.161177] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4384.167470] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4384.173200] ? watchdog+0x40/0x40 [ 4384.174083] ? hrtimer_interrupt+0x128/0x2c0 [ 4384.175246] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4384.184888] ? apic_timer_interrupt+0xf/0x20 [ 4384.186042] [ 4384.191217] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4384.194240] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4384.197511] do_raw_spin_lock+0x177/0x190 [ 4384.200087] _raw_spin_lock+0x21/0x30 [ 4384.203862] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4384.206104] ? do_raw_spin_unlock+0x75/0x190 [ 4384.210351] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4384.216182] ? do_wait_intr+0xf0/0xf0 [ 4384.219821] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4384.223977] mdc_reint+0x2d/0x110 [mdc] [ 4384.226849] mdc_unlink+0x658/0x870 [mdc] [ 4384.230007] lmv_unlink+0x34d/0xbd0 [lmv] [ 4384.231881] ll_unlink+0x24d/0x7f0 [lustre] [ 4384.233572] vfs_unlink+0x202/0x330 [ 4384.235191] do_unlinkat+0x2fa/0x460 [ 4384.238078] __x64_sys_unlink+0x37/0x50 [ 4384.239549] do_syscall_64+0xc1/0x440 [ 4384.241625] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4384.243566] RIP: 0033:0x7f83e5f5c10b [ 4384.245058] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4384.257263] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4384.269251] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4384.276009] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4384.279133] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4384.283343] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4384.285827] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4400.060023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 4400.066102] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4400.093310] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4400.100513] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4400.105383] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4400.109248] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4400.112377] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4400.121969] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4400.126290] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4400.130409] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4400.137664] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4400.143509] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4400.147289] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4400.152982] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4400.158393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4400.161402] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4400.165242] Call Trace: [ 4400.166646] [ 4400.168060] ? show_regs.cold.9+0x22/0x2f [ 4400.170244] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4400.184448] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4400.186451] ? watchdog+0x40/0x40 [ 4400.188355] ? hrtimer_interrupt+0x128/0x2c0 [ 4400.191590] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4400.196983] ? apic_timer_interrupt+0xf/0x20 [ 4400.199092] [ 4400.200373] ? native_safe_halt+0x2e/0x50 [ 4400.202652] kvm_wait+0xcd/0xe0 [ 4400.204473] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4400.207230] do_raw_spin_lock+0x177/0x190 [ 4400.209327] _raw_spin_lock+0x21/0x30 [ 4400.210808] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4400.214288] process_one_work+0x2c8/0x700 [ 4400.216486] worker_thread+0x296/0x6e0 [ 4400.218291] ? rescuer_thread+0x570/0x570 [ 4400.220292] kthread+0x1d1/0x200 [ 4400.221966] ? set_kthread_struct+0x70/0x70 [ 4400.224299] ret_from_fork+0x1f/0x30 [ 4401.634315] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1888s! [ 4401.640179] Showing busy workqueues and worker pools: [ 4401.643728] workqueue events: flags=0x0 [ 4401.645685] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4401.650130] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4401.660030] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4401.671799] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4401.676443] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4401.683703] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4401.695294] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4401.700331] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4401.706612] workqueue events_long: flags=0x0 [ 4401.713047] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4401.723046] pending: lu_objects_flush [obdclass] [ 4401.735837] workqueue events_unbound: flags=0x2 [ 4401.741547] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4401.747360] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4401.752471] workqueue events_power_efficient: flags=0x80 [ 4401.759480] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4401.764180] pending: neigh_periodic_work, do_cache_clean [ 4401.771868] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4401.782120] pending: neigh_periodic_work, check_lifetime [ 4401.785808] workqueue mm_percpu_wq: flags=0x8 [ 4401.795840] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4401.806588] pending: vmstat_update, lru_add_drain_per_cpu [ 4401.815822] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4401.833267] pending: vmstat_update [ 4401.835631] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4401.844824] pending: lru_add_drain_per_cpu BAR(48) [ 4401.853257] workqueue ipv6_addrconf: flags=0x40008 [ 4401.858736] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4401.868512] pending: addrconf_verify_work [ 4401.875056] workqueue knbd0-recv: flags=0x1a [ 4401.881498] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4401.887839] in-flight: 650:recv_work [ 4401.889828] workqueue ptlrpc_pinger: flags=0x2 [ 4401.892858] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4401.896715] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4401.901112] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1888s workers=4 idle: 36 466 29757 [ 4401.910586] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4391s workers=2 idle: 652 [ 4408.059391] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 4408.063218] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4408.079760] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4408.084069] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4408.088056] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4408.090355] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4408.098653] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4408.101066] RAX: 0000000000002133 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4408.103387] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4408.106112] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4408.108862] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4408.111418] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4408.114295] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4408.117304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4408.120072] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4408.122718] Call Trace: [ 4408.123355] [ 4408.124153] ? show_regs.cold.9+0x22/0x2f [ 4408.125690] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4408.127153] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4408.128700] ? watchdog+0x40/0x40 [ 4408.130665] ? hrtimer_interrupt+0x128/0x2c0 [ 4408.132095] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4408.133419] ? apic_timer_interrupt+0xf/0x20 [ 4408.134946] [ 4408.135639] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4408.137236] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4408.139330] do_raw_spin_lock+0x177/0x190 [ 4408.141502] _raw_spin_lock+0x21/0x30 [ 4408.143508] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4408.146721] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4408.148752] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4408.150378] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4408.152790] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4408.154864] ? woken_wake_function+0x30/0x30 [ 4408.158125] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4408.160207] kthread+0x1d1/0x200 [ 4408.161774] ? set_kthread_struct+0x70/0x70 [ 4408.162990] ret_from_fork+0x1f/0x30 [ 4412.070468] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 4412.077886] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4412.126671] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4412.135870] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4412.142479] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4412.148882] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4412.167892] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4412.178936] RAX: 0000000000005168 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4412.188969] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4412.204871] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4412.217523] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4412.227678] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4412.236677] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4412.249426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4412.259979] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4412.267009] Call Trace: [ 4412.271360] [ 4412.272042] ? show_regs.cold.9+0x22/0x2f [ 4412.275079] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4412.278482] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4412.281716] ? watchdog+0x40/0x40 [ 4412.285696] ? hrtimer_interrupt+0x128/0x2c0 [ 4412.291534] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4412.294615] ? apic_timer_interrupt+0xf/0x20 [ 4412.299429] [ 4412.300084] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4412.315564] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4412.323711] do_raw_spin_lock+0x177/0x190 [ 4412.327859] _raw_spin_lock+0x21/0x30 [ 4412.335958] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4412.344030] ? do_raw_spin_unlock+0x75/0x190 [ 4412.351838] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4412.359335] ? do_wait_intr+0xf0/0xf0 [ 4412.368733] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4412.374917] mdc_reint+0x2d/0x110 [mdc] [ 4412.382962] mdc_unlink+0x658/0x870 [mdc] [ 4412.386646] lmv_unlink+0x34d/0xbd0 [lmv] [ 4412.392966] ll_unlink+0x24d/0x7f0 [lustre] [ 4412.398340] vfs_unlink+0x202/0x330 [ 4412.401504] do_unlinkat+0x2fa/0x460 [ 4412.410343] __x64_sys_unlink+0x37/0x50 [ 4412.417577] do_syscall_64+0xc1/0x440 [ 4412.421806] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4412.428935] RIP: 0033:0x7f83e5f5c10b [ 4412.435611] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4412.469246] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4412.478959] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4412.490892] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4412.501806] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4412.514522] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4412.525064] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4428.061189] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 4428.061505] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 4428.066488] Modules linked in: lustre(O) [ 4428.068970] Modules linked in: [ 4428.068979] osp(O) [ 4428.070076] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss [ 4428.072018] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) [ 4428.072596] nfsv4 [ 4428.083942] ptlrpc(O) [ 4428.090566] dns_resolver [ 4428.091821] obdclass(O) [ 4428.093028] intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 [ 4428.094526] ksocklnd(O) [ 4428.101416] squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4428.101439] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4428.101444] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4428.101449] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4428.101680] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 4428.101688] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 4428.101691] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4428.101694] RAX: 000000000000423f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4428.101696] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4428.101698] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4428.101701] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4428.101703] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4428.101706] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4428.101709] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4428.101711] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4428.101720] Call Trace: [ 4428.101725] [ 4428.101730] ? show_regs.cold.9+0x22/0x2f [ 4428.101739] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4428.101746] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4428.101751] ? watchdog+0x40/0x40 [ 4428.101754] ? hrtimer_interrupt+0x128/0x2c0 [ 4428.101756] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4428.101762] ? apic_timer_interrupt+0xf/0x20 [ 4428.101765] [ 4428.101767] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 4428.101771] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4428.101774] do_raw_spin_lock+0x177/0x190 [ 4428.101778] _raw_spin_lock+0x21/0x30 [ 4428.101781] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4428.101914] process_one_work+0x2c8/0x700 [ 4428.101918] worker_thread+0x296/0x6e0 [ 4428.101921] ? rescuer_thread+0x570/0x570 [ 4428.101923] kthread+0x1d1/0x200 [ 4428.101926] ? set_kthread_struct+0x70/0x70 [ 4428.101928] ret_from_fork+0x1f/0x30 [ 4428.224651] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4428.236775] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4428.240667] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4428.244134] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4428.246040] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4428.254497] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4428.257726] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4428.261608] RDX: 0000000000006f0e RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4428.265552] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4428.268913] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4428.272118] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4428.276118] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4428.281380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4428.283822] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4428.286867] Call Trace: [ 4428.288020] [ 4428.289213] ? show_regs.cold.9+0x22/0x2f [ 4428.291141] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4428.294399] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4428.296832] ? watchdog+0x40/0x40 [ 4428.300903] ? hrtimer_interrupt+0x128/0x2c0 [ 4428.302718] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4428.307485] ? apic_timer_interrupt+0xf/0x20 [ 4428.309702] [ 4428.310759] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4428.315013] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4428.319465] do_raw_spin_lock+0x177/0x190 [ 4428.322912] _raw_spin_lock+0x21/0x30 [ 4428.325513] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4428.328934] obd_mod_free+0x46/0x140 [ptlrpc] [ 4428.331956] mdc_commit_open+0x91/0xb0 [mdc] [ 4428.335537] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4428.338935] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4428.342441] after_reply+0xe2c/0x1780 [ptlrpc] [ 4428.345833] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4428.349729] ? __next_timer_interrupt+0x160/0x160 [ 4428.352432] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4428.354826] ? do_wait_intr+0xf0/0xf0 [ 4428.357943] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4428.361766] kthread+0x1d1/0x200 [ 4428.364313] ? set_kthread_struct+0x70/0x70 [ 4428.367706] ret_from_fork+0x1f/0x30 [ 4432.355807] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1919s! [ 4432.360662] Showing busy workqueues and worker pools: [ 4432.372747] workqueue events: flags=0x0 [ 4432.375534] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4432.386626] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4432.402553] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4432.408530] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4432.415687] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4432.427025] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4432.435513] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4432.444645] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4432.452965] workqueue events_long: flags=0x0 [ 4432.455526] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4432.459242] pending: lu_objects_flush [obdclass] [ 4432.465979] workqueue events_unbound: flags=0x2 [ 4432.468479] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4432.472056] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4432.477716] workqueue events_power_efficient: flags=0x80 [ 4432.480442] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4432.485575] pending: neigh_periodic_work, do_cache_clean [ 4432.488621] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4432.494000] pending: neigh_periodic_work, check_lifetime [ 4432.496937] workqueue mm_percpu_wq: flags=0x8 [ 4432.499351] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4432.503175] pending: vmstat_update, lru_add_drain_per_cpu [ 4432.510390] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4432.517170] pending: vmstat_update [ 4432.520611] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4432.528562] pending: lru_add_drain_per_cpu BAR(48) [ 4432.534834] workqueue ipv6_addrconf: flags=0x40008 [ 4432.537898] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4432.541631] pending: addrconf_verify_work [ 4432.550453] workqueue knbd0-recv: flags=0x1a [ 4432.553119] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4432.560125] in-flight: 650:recv_work [ 4432.564121] workqueue ptlrpc_pinger: flags=0x2 [ 4432.568094] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4432.571317] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4432.575713] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1919s workers=4 idle: 36 466 29757 [ 4432.580978] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4422s workers=2 idle: 652 [ 4436.059550] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 4436.064737] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4436.083490] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4436.089503] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4436.093143] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4436.095350] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4436.102663] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4436.105527] RAX: 0000000000003a88 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4436.108161] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4436.111208] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4436.113612] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4436.116436] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4436.119218] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4436.122586] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4436.125163] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4436.127856] Call Trace: [ 4436.129443] [ 4436.130261] ? show_regs.cold.9+0x22/0x2f [ 4436.131676] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4436.133344] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4436.134845] ? watchdog+0x40/0x40 [ 4436.136308] ? hrtimer_interrupt+0x128/0x2c0 [ 4436.138046] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4436.139761] ? apic_timer_interrupt+0xf/0x20 [ 4436.141249] [ 4436.141848] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4436.144066] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4436.145773] do_raw_spin_lock+0x177/0x190 [ 4436.147487] _raw_spin_lock+0x21/0x30 [ 4436.148964] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4436.151913] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4436.153856] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4436.156456] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4436.158752] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4436.162097] ? woken_wake_function+0x30/0x30 [ 4436.163747] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4436.166170] kthread+0x1d1/0x200 [ 4436.167227] ? set_kthread_struct+0x70/0x70 [ 4436.168500] ret_from_fork+0x1f/0x30 [ 4440.062010] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 4440.065333] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4440.084801] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4440.089773] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4440.093726] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4440.096201] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4440.104047] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4440.107037] RAX: 000000000000274d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4440.110042] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4440.112951] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4440.116070] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4440.119349] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4440.122659] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4440.126117] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4440.128556] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4440.130938] Call Trace: [ 4440.131808] [ 4440.132736] ? show_regs.cold.9+0x22/0x2f [ 4440.134515] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4440.136602] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4440.138423] ? watchdog+0x40/0x40 [ 4440.139943] ? hrtimer_interrupt+0x128/0x2c0 [ 4440.142877] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4440.144844] ? apic_timer_interrupt+0xf/0x20 [ 4440.147947] [ 4440.148919] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4440.151240] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4440.154010] do_raw_spin_lock+0x177/0x190 [ 4440.155745] _raw_spin_lock+0x21/0x30 [ 4440.157483] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4440.160579] ? do_raw_spin_unlock+0x75/0x190 [ 4440.162354] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4440.164609] ? do_wait_intr+0xf0/0xf0 [ 4440.166223] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4440.168331] mdc_reint+0x2d/0x110 [mdc] [ 4440.170019] mdc_unlink+0x658/0x870 [mdc] [ 4440.171715] lmv_unlink+0x34d/0xbd0 [lmv] [ 4440.173481] ll_unlink+0x24d/0x7f0 [lustre] [ 4440.175215] vfs_unlink+0x202/0x330 [ 4440.176948] do_unlinkat+0x2fa/0x460 [ 4440.178522] __x64_sys_unlink+0x37/0x50 [ 4440.180318] do_syscall_64+0xc1/0x440 [ 4440.181938] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4440.184192] RIP: 0033:0x7f83e5f5c10b [ 4440.185780] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4440.193590] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4440.196907] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4440.199980] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4440.202111] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4440.204704] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4440.208092] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4456.062575] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 4456.077242] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4456.123343] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4456.140785] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4456.152762] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4456.165644] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4456.171436] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4456.190885] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4456.201084] RAX: 000000000000199b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4456.205825] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4456.210180] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4456.214360] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4456.222288] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4456.227718] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4456.232519] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4456.235498] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4456.240190] Call Trace: [ 4456.241202] [ 4456.241790] ? show_regs.cold.9+0x22/0x2f [ 4456.243561] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4456.245404] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4456.247364] ? watchdog+0x40/0x40 [ 4456.248618] ? hrtimer_interrupt+0x128/0x2c0 [ 4456.249965] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4456.252031] ? apic_timer_interrupt+0xf/0x20 [ 4456.253609] [ 4456.254502] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4456.256671] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4456.258597] do_raw_spin_lock+0x177/0x190 [ 4456.260035] _raw_spin_lock+0x21/0x30 [ 4456.267811] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4456.271098] process_one_work+0x2c8/0x700 [ 4456.274878] worker_thread+0x296/0x6e0 [ 4456.278806] ? rescuer_thread+0x570/0x570 [ 4456.280424] kthread+0x1d1/0x200 [ 4456.281738] ? set_kthread_struct+0x70/0x70 [ 4456.283428] ret_from_fork+0x1f/0x30 [ 4460.056037] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 4460.058824] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4460.086745] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4460.091762] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4460.097318] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4460.100093] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4460.113363] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4460.117522] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4460.121153] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4460.125134] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4460.128699] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4460.132307] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4460.136097] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4460.140152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4460.142822] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4460.146268] Call Trace: [ 4460.147826] [ 4460.149311] ? show_regs.cold.9+0x22/0x2f [ 4460.151608] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4460.153746] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4460.156310] ? watchdog+0x40/0x40 [ 4460.161458] ? hrtimer_interrupt+0x128/0x2c0 [ 4460.166043] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4460.167913] ? apic_timer_interrupt+0xf/0x20 [ 4460.172448] [ 4460.174141] ? native_safe_halt+0x2e/0x50 [ 4460.177543] kvm_wait+0xcd/0xe0 [ 4460.181927] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4460.185591] do_raw_spin_lock+0x177/0x190 [ 4460.190568] _raw_spin_lock+0x21/0x30 [ 4460.192801] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4460.198811] obd_mod_free+0x46/0x140 [ptlrpc] [ 4460.203491] mdc_commit_open+0x91/0xb0 [mdc] [ 4460.206595] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4460.211506] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4460.217352] after_reply+0xe2c/0x1780 [ptlrpc] [ 4460.222109] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4460.227296] ? __next_timer_interrupt+0x160/0x160 [ 4460.231645] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4460.235155] ? do_wait_intr+0xf0/0xf0 [ 4460.237677] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4460.243822] kthread+0x1d1/0x200 [ 4460.245854] ? set_kthread_struct+0x70/0x70 [ 4460.248112] ret_from_fork+0x1f/0x30 [ 4463.072770] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1949s! [ 4463.081581] Showing busy workqueues and worker pools: [ 4463.084640] workqueue events: flags=0x0 [ 4463.091332] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4463.097199] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4463.113119] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4463.125109] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4463.129053] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4463.136429] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4463.141762] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4463.144440] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4463.151545] workqueue events_long: flags=0x0 [ 4463.154796] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4463.158564] pending: lu_objects_flush [obdclass] [ 4463.161332] workqueue events_unbound: flags=0x2 [ 4463.163517] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4463.166617] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4463.171639] workqueue events_power_efficient: flags=0x80 [ 4463.175759] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4463.180617] pending: neigh_periodic_work, do_cache_clean [ 4463.183823] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4463.192819] pending: neigh_periodic_work, check_lifetime [ 4463.199687] workqueue mm_percpu_wq: flags=0x8 [ 4463.205910] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4463.219435] pending: vmstat_update, lru_add_drain_per_cpu [ 4463.234549] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4463.237944] pending: vmstat_update [ 4463.238980] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4463.241469] pending: lru_add_drain_per_cpu BAR(48) [ 4463.243564] workqueue ipv6_addrconf: flags=0x40008 [ 4463.245186] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4463.247583] pending: addrconf_verify_work [ 4463.249071] workqueue knbd0-recv: flags=0x1a [ 4463.251271] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4463.254599] in-flight: 650:recv_work [ 4463.256007] workqueue ptlrpc_pinger: flags=0x2 [ 4463.258202] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4463.260609] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4463.263551] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1950s workers=4 idle: 36 466 29757 [ 4463.267146] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4453s workers=2 idle: 652 [ 4464.059014] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 4464.064603] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4464.127717] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4464.140297] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4464.147315] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4464.153214] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4464.165994] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4464.177431] RAX: 000000000000548d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4464.183161] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4464.188322] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4464.191632] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4464.197112] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4464.203591] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4464.210644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4464.212538] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4464.218825] Call Trace: [ 4464.220102] [ 4464.221162] ? show_regs.cold.9+0x22/0x2f [ 4464.225568] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4464.229815] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4464.231373] ? watchdog+0x40/0x40 [ 4464.233504] ? hrtimer_interrupt+0x128/0x2c0 [ 4464.235198] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4464.239326] ? apic_timer_interrupt+0xf/0x20 [ 4464.241985] [ 4464.243190] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4464.246494] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4464.251166] do_raw_spin_lock+0x177/0x190 [ 4464.253835] _raw_spin_lock+0x21/0x30 [ 4464.256061] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4464.259233] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4464.261678] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4464.263260] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4464.275279] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4464.278235] ? woken_wake_function+0x30/0x30 [ 4464.281288] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4464.285206] kthread+0x1d1/0x200 [ 4464.287147] ? set_kthread_struct+0x70/0x70 [ 4464.291414] ret_from_fork+0x1f/0x30 [ 4468.062298] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 4468.073563] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4468.138901] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4468.151491] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4468.170119] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4468.179715] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4468.209369] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4468.217784] RAX: 0000000000005e2b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4468.239964] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4468.259774] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4468.268489] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4468.272397] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4468.276353] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4468.280017] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4468.282985] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4468.286289] Call Trace: [ 4468.288152] [ 4468.289498] ? show_regs.cold.9+0x22/0x2f [ 4468.292233] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4468.297568] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4468.303757] ? watchdog+0x40/0x40 [ 4468.304985] ? hrtimer_interrupt+0x128/0x2c0 [ 4468.306535] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4468.313038] ? apic_timer_interrupt+0xf/0x20 [ 4468.321026] [ 4468.322097] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4468.328014] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4468.334006] do_raw_spin_lock+0x177/0x190 [ 4468.337732] _raw_spin_lock+0x21/0x30 [ 4468.353334] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4468.355325] ? do_raw_spin_unlock+0x75/0x190 [ 4468.370776] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4468.380787] ? do_wait_intr+0xf0/0xf0 [ 4468.383722] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4468.396337] mdc_reint+0x2d/0x110 [mdc] [ 4468.400213] mdc_unlink+0x658/0x870 [mdc] [ 4468.408311] lmv_unlink+0x34d/0xbd0 [lmv] [ 4468.416024] ll_unlink+0x24d/0x7f0 [lustre] [ 4468.417827] vfs_unlink+0x202/0x330 [ 4468.425378] do_unlinkat+0x2fa/0x460 [ 4468.429661] __x64_sys_unlink+0x37/0x50 [ 4468.435397] do_syscall_64+0xc1/0x440 [ 4468.444350] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4468.454374] RIP: 0033:0x7f83e5f5c10b [ 4468.459019] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4468.490614] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4468.504803] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4468.517106] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4468.523403] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4468.528677] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4468.537011] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4484.060166] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 4484.067787] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4484.110811] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4484.115291] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4484.120755] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4484.125294] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4484.132249] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4484.151085] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4484.155859] RAX: 000000000000679e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4484.164265] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4484.167857] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4484.170694] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4484.173943] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4484.181069] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4484.189116] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4484.192216] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4484.202092] Call Trace: [ 4484.203560] [ 4484.204835] ? show_regs.cold.9+0x22/0x2f [ 4484.207644] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4484.210586] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4484.213525] ? watchdog+0x40/0x40 [ 4484.215960] ? hrtimer_interrupt+0x128/0x2c0 [ 4484.225767] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4484.228402] ? apic_timer_interrupt+0xf/0x20 [ 4484.230301] [ 4484.231368] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4484.233608] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4484.236520] do_raw_spin_lock+0x177/0x190 [ 4484.240553] _raw_spin_lock+0x21/0x30 [ 4484.241933] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4484.249849] process_one_work+0x2c8/0x700 [ 4484.253364] worker_thread+0x296/0x6e0 [ 4484.255772] ? rescuer_thread+0x570/0x570 [ 4484.257230] kthread+0x1d1/0x200 [ 4484.261024] ? set_kthread_struct+0x70/0x70 [ 4484.263194] ret_from_fork+0x1f/0x30 [ 4488.056053] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 4488.060323] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4488.097876] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4488.107531] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4488.115314] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4488.119355] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4488.145174] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4488.162504] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4488.175579] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4488.178054] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4488.200210] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4488.209831] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4488.217869] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4488.226645] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4488.232129] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4488.245560] Call Trace: [ 4488.248196] [ 4488.249535] ? show_regs.cold.9+0x22/0x2f [ 4488.254644] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4488.260434] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4488.263121] ? watchdog+0x40/0x40 [ 4488.264853] ? hrtimer_interrupt+0x128/0x2c0 [ 4488.268505] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4488.274209] ? apic_timer_interrupt+0xf/0x20 [ 4488.284668] [ 4488.289691] ? native_safe_halt+0x2e/0x50 [ 4488.300652] kvm_wait+0xcd/0xe0 [ 4488.310814] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4488.315702] do_raw_spin_lock+0x177/0x190 [ 4488.321218] _raw_spin_lock+0x21/0x30 [ 4488.326604] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4488.334201] obd_mod_free+0x46/0x140 [ptlrpc] [ 4488.341134] mdc_commit_open+0x91/0xb0 [mdc] [ 4488.346302] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4488.352284] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4488.356899] after_reply+0xe2c/0x1780 [ptlrpc] [ 4488.365099] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4488.375620] ? __next_timer_interrupt+0x160/0x160 [ 4488.377326] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4488.389773] ? do_wait_intr+0xf0/0xf0 [ 4488.394141] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4488.403360] kthread+0x1d1/0x200 [ 4488.412018] ? set_kthread_struct+0x70/0x70 [ 4488.413552] ret_from_fork+0x1f/0x30 [ 4492.062669] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 4492.067267] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4492.091542] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4492.096938] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4492.101590] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4492.104338] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4492.114126] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4492.117582] RAX: 0000000000007289 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4492.121413] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4492.125056] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4492.132000] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4492.135845] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4492.140533] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4492.148542] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4492.154361] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4492.161587] Call Trace: [ 4492.162816] [ 4492.164085] ? show_regs.cold.9+0x22/0x2f [ 4492.169611] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4492.174016] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4492.176301] ? watchdog+0x40/0x40 [ 4492.179554] ? hrtimer_interrupt+0x128/0x2c0 [ 4492.181981] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4492.186623] ? apic_timer_interrupt+0xf/0x20 [ 4492.189019] [ 4492.191219] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4492.194506] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4492.196947] do_raw_spin_lock+0x177/0x190 [ 4492.199207] _raw_spin_lock+0x21/0x30 [ 4492.204549] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4492.208828] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4492.212385] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4492.216530] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4492.219130] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4492.222200] ? woken_wake_function+0x30/0x30 [ 4492.224726] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4492.228819] kthread+0x1d1/0x200 [ 4492.230571] ? set_kthread_struct+0x70/0x70 [ 4492.232292] ret_from_fork+0x1f/0x30 [ 4493.284927] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1980s! [ 4493.288688] Showing busy workqueues and worker pools: [ 4493.293095] workqueue events: flags=0x0 [ 4493.294472] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4493.297802] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4493.303046] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4493.306978] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4493.310300] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4493.313990] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4493.318880] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4493.325833] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4493.337518] workqueue events_long: flags=0x0 [ 4493.341941] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4493.355440] pending: lu_objects_flush [obdclass] [ 4493.362696] workqueue events_unbound: flags=0x2 [ 4493.367392] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4493.375926] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4493.385961] workqueue events_power_efficient: flags=0x80 [ 4493.393400] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4493.401511] pending: neigh_periodic_work, do_cache_clean [ 4493.404965] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4493.415442] pending: neigh_periodic_work, check_lifetime [ 4493.422689] workqueue mm_percpu_wq: flags=0x8 [ 4493.430240] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4493.440097] pending: vmstat_update, lru_add_drain_per_cpu [ 4493.447911] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4493.460921] pending: vmstat_update [ 4493.469190] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4493.474915] pending: lru_add_drain_per_cpu BAR(48) [ 4493.490234] workqueue ipv6_addrconf: flags=0x40008 [ 4493.498271] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4493.501532] pending: addrconf_verify_work [ 4493.506529] workqueue knbd0-recv: flags=0x1a [ 4493.510587] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4493.516032] in-flight: 650:recv_work [ 4493.518535] workqueue ptlrpc_pinger: flags=0x2 [ 4493.525854] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4493.527721] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4493.535109] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1980s workers=4 idle: 36 466 29757 [ 4493.545873] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4483s workers=2 idle: 652 [ 4496.062011] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 4496.065126] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4496.087156] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4496.094154] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4496.098170] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4496.100317] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4496.109127] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4496.113820] RAX: 0000000000005a1e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4496.117951] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4496.121143] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4496.124090] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4496.126900] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4496.130789] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4496.134428] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4496.138198] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4496.140256] Call Trace: [ 4496.141171] [ 4496.145302] ? show_regs.cold.9+0x22/0x2f [ 4496.151501] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4496.153333] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4496.154977] ? watchdog+0x40/0x40 [ 4496.157073] ? hrtimer_interrupt+0x128/0x2c0 [ 4496.160248] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4496.164143] ? apic_timer_interrupt+0xf/0x20 [ 4496.167751] [ 4496.168206] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4496.170714] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4496.173223] do_raw_spin_lock+0x177/0x190 [ 4496.175217] _raw_spin_lock+0x21/0x30 [ 4496.176445] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4496.180005] ? do_raw_spin_unlock+0x75/0x190 [ 4496.182946] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4496.185712] ? do_wait_intr+0xf0/0xf0 [ 4496.190286] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4496.199045] mdc_reint+0x2d/0x110 [mdc] [ 4496.205022] mdc_unlink+0x658/0x870 [mdc] [ 4496.208928] lmv_unlink+0x34d/0xbd0 [lmv] [ 4496.217803] ll_unlink+0x24d/0x7f0 [lustre] [ 4496.225488] vfs_unlink+0x202/0x330 [ 4496.226643] do_unlinkat+0x2fa/0x460 [ 4496.227502] __x64_sys_unlink+0x37/0x50 [ 4496.231879] do_syscall_64+0xc1/0x440 [ 4496.233248] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4496.236186] RIP: 0033:0x7f83e5f5c10b [ 4496.240087] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4496.253879] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4496.259855] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4496.267006] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4496.270933] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4496.274368] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4496.282780] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4512.060240] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 4512.064672] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4512.095488] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4512.102986] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4512.106874] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4512.110543] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4512.113534] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4512.124865] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4512.128559] RAX: 0000000000007d0f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4512.132283] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4512.137115] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4512.140958] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4512.144596] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4512.147770] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4512.152309] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4512.155162] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4512.158770] Call Trace: [ 4512.160037] [ 4512.161170] ? show_regs.cold.9+0x22/0x2f [ 4512.162838] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4512.165195] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4512.166895] ? watchdog+0x40/0x40 [ 4512.168885] ? hrtimer_interrupt+0x128/0x2c0 [ 4512.171301] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4512.173684] ? apic_timer_interrupt+0xf/0x20 [ 4512.175583] [ 4512.176821] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4512.179538] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4512.181671] do_raw_spin_lock+0x177/0x190 [ 4512.183799] _raw_spin_lock+0x21/0x30 [ 4512.185469] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4512.188365] process_one_work+0x2c8/0x700 [ 4512.190303] worker_thread+0x296/0x6e0 [ 4512.192304] ? rescuer_thread+0x570/0x570 [ 4512.194715] kthread+0x1d1/0x200 [ 4512.196210] ? set_kthread_struct+0x70/0x70 [ 4512.198407] ret_from_fork+0x1f/0x30 [ 4520.056060] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 4520.058697] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4520.059387] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 4520.087577] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4520.089958] Modules linked in: [ 4520.094866] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4520.094868] lustre(O) [ 4520.094938] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4520.096006] osp(O) [ 4520.099737] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4520.100851] ofd(O) [ 4520.102774] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 [ 4520.103583] lod(O) [ 4520.110559] ORIG_RAX: ffffffffffffff13 [ 4520.110572] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4520.110575] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4520.110577] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4520.111260] mdt(O) [ 4520.115228] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4520.115866] mdd(O) [ 4520.117697] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4520.117705] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4520.117707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4520.117709] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4520.117718] Call Trace: [ 4520.117724] [ 4520.117729] ? show_regs.cold.9+0x22/0x2f [ 4520.117742] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4520.117748] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4520.117752] ? watchdog+0x40/0x40 [ 4520.117756] ? hrtimer_interrupt+0x128/0x2c0 [ 4520.117759] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4520.117766] ? apic_timer_interrupt+0xf/0x20 [ 4520.117770] [ 4520.117773] ? native_safe_halt+0x2e/0x50 [ 4520.117777] kvm_wait+0xcd/0xe0 [ 4520.117846] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4520.117854] do_raw_spin_lock+0x177/0x190 [ 4520.117858] _raw_spin_lock+0x21/0x30 [ 4520.117863] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4520.121041] mgs(O) [ 4520.124074] obd_mod_free+0x46/0x140 [ptlrpc] [ 4520.127872] lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) [ 4520.128757] mdc_commit_open+0x91/0xb0 [mdc] [ 4520.131790] lov(O) [ 4520.132402] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4520.135161] osc(O) lmv(O) ec(O) fid(O) [ 4520.138301] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4520.141003] fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4520.141056] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4520.141062] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4520.141065] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4520.141078] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4520.141082] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4520.141087] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4520.141089] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4520.141091] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4520.141093] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4520.141095] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4520.141098] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4520.141101] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4520.141113] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4520.141120] Call Trace: [ 4520.141125] [ 4520.141130] ? show_regs.cold.9+0x22/0x2f [ 4520.141140] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4520.141146] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4520.141149] ? watchdog+0x40/0x40 [ 4520.141154] ? hrtimer_interrupt+0x128/0x2c0 [ 4520.141158] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4520.141164] ? apic_timer_interrupt+0xf/0x20 [ 4520.141168] [ 4520.141171] ? native_safe_halt+0x2e/0x50 [ 4520.141175] kvm_wait+0xcd/0xe0 [ 4520.141182] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4520.141189] do_raw_spin_lock+0x177/0x190 [ 4520.141194] _raw_spin_lock+0x21/0x30 [ 4520.141199] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4520.141460] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4520.142002] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4520.142250] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4520.142487] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4520.143005] ? woken_wake_function+0x30/0x30 [ 4520.143015] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4520.143238] kthread+0x1d1/0x200 [ 4520.143244] ? set_kthread_struct+0x70/0x70 [ 4520.143247] ret_from_fork+0x1f/0x30 [ 4520.397061] after_reply+0xe2c/0x1780 [ptlrpc] [ 4520.408557] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4520.424605] ? __next_timer_interrupt+0x160/0x160 [ 4520.426414] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4520.431847] ? do_wait_intr+0xf0/0xf0 [ 4520.433707] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4520.436090] kthread+0x1d1/0x200 [ 4520.438470] ? set_kthread_struct+0x70/0x70 [ 4520.444196] ret_from_fork+0x1f/0x30 [ 4524.000724] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2010s! [ 4524.009144] Showing busy workqueues and worker pools: [ 4524.012898] workqueue events: flags=0x0 [ 4524.020770] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4524.028554] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4524.033689] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4524.037383] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4524.042381] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4524.047760] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4524.052188] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4524.056836] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4524.062090] workqueue events_long: flags=0x0 [ 4524.062259] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4524.064839] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4524.064857] pending: lu_objects_flush [obdclass] [ 4524.065040] workqueue events_unbound: flags=0x2 [ 4524.065044] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4524.065051] pending: flush_memcg_stats_dwork [ 4524.069554] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4524.069635] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4524.069648] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4524.069651] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4524.069665] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4524.069669] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4524.069672] RAX: 000000000000608f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4524.069675] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4524.069677] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4524.069679] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4524.069682] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4524.069685] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4524.069688] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4524.069690] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4524.069699] Call Trace: [ 4524.073486] , idle_cull_fn [ 4524.073529] workqueue events_power_efficient: flags=0x80 [ 4524.073534] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4524.073546] pending: neigh_periodic_work, do_cache_clean [ 4524.073561] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4524.073570] pending: neigh_periodic_work, check_lifetime [ 4524.073584] workqueue mm_percpu_wq: flags=0x8 [ 4524.073587] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4524.073593] pending: vmstat_update, lru_add_drain_per_cpu [ 4524.073605] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4524.073613] pending: vmstat_update [ 4524.073618] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4524.073625] pending: lru_add_drain_per_cpu BAR(48) [ 4524.073650] workqueue ipv6_addrconf: flags=0x40008 [ 4524.073653] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4524.073659] pending: addrconf_verify_work [ 4524.073672] workqueue knbd0-recv: flags=0x1a [ 4524.073675] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4524.073681] in-flight: 650:recv_work [ 4524.073690] workqueue ptlrpc_pinger: flags=0x2 [ 4524.073693] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4524.073699] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4524.081120] [ 4524.081138] ? show_regs.cold.9+0x22/0x2f [ 4524.081152] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4524.081159] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4524.081167] ? watchdog+0x40/0x40 [ 4524.081171] ? hrtimer_interrupt+0x128/0x2c0 [ 4524.081176] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4524.081185] ? apic_timer_interrupt+0xf/0x20 [ 4524.081189] [ 4524.081193] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4524.081199] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4524.081203] do_raw_spin_lock+0x177/0x190 [ 4524.081207] _raw_spin_lock+0x21/0x30 [ 4524.081214] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4524.081974] ? do_raw_spin_unlock+0x75/0x190 [ 4524.081981] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4524.087769] [ 4524.087856] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2010s workers=4 idle: 36 466 29757 [ 4524.087879] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4514s workers=2 idle: 652 [ 4524.349543] ? do_wait_intr+0xf0/0xf0 [ 4524.354921] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4524.357675] mdc_reint+0x2d/0x110 [mdc] [ 4524.359327] mdc_unlink+0x658/0x870 [mdc] [ 4524.360644] lmv_unlink+0x34d/0xbd0 [lmv] [ 4524.362464] ll_unlink+0x24d/0x7f0 [lustre] [ 4524.366161] vfs_unlink+0x202/0x330 [ 4524.368383] do_unlinkat+0x2fa/0x460 [ 4524.370525] __x64_sys_unlink+0x37/0x50 [ 4524.372036] do_syscall_64+0xc1/0x440 [ 4524.376411] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4524.378235] RIP: 0033:0x7f83e5f5c10b [ 4524.379581] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4524.391767] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4524.399105] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4524.401875] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4524.404706] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4524.412420] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4524.415017] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4540.060100] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 4540.063035] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4540.082855] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4540.087420] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4540.091193] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4540.094238] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4540.096944] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4540.105972] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4540.119374] RAX: 0000000000000513 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4540.126053] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4540.131632] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4540.138778] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4540.144907] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4540.152066] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4540.156921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4540.165727] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4540.176791] Call Trace: [ 4540.178004] [ 4540.178937] ? show_regs.cold.9+0x22/0x2f [ 4540.184003] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4540.190085] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4540.192981] ? watchdog+0x40/0x40 [ 4540.197641] ? hrtimer_interrupt+0x128/0x2c0 [ 4540.202062] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4540.207950] ? apic_timer_interrupt+0xf/0x20 [ 4540.213893] [ 4540.217297] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4540.220243] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4540.222904] do_raw_spin_lock+0x177/0x190 [ 4540.224699] _raw_spin_lock+0x21/0x30 [ 4540.226503] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4540.229100] process_one_work+0x2c8/0x700 [ 4540.231428] worker_thread+0x296/0x6e0 [ 4540.232970] ? rescuer_thread+0x570/0x570 [ 4540.235204] kthread+0x1d1/0x200 [ 4540.236814] ? set_kthread_struct+0x70/0x70 [ 4540.241519] ret_from_fork+0x1f/0x30 [ 4548.059183] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 4548.063112] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4548.086744] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4548.092633] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4548.097014] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4548.100232] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4548.110317] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4548.114045] RAX: 0000000000004ecc RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4548.118373] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4548.122539] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4548.126811] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4548.131350] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4548.134689] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4548.139516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4548.142979] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4548.147108] Call Trace: [ 4548.148567] [ 4548.149630] ? show_regs.cold.9+0x22/0x2f [ 4548.152181] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4548.155965] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4548.158861] ? watchdog+0x40/0x40 [ 4548.161295] ? hrtimer_interrupt+0x128/0x2c0 [ 4548.163629] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4548.166397] ? apic_timer_interrupt+0xf/0x20 [ 4548.168759] [ 4548.170142] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4548.173467] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4548.176643] do_raw_spin_lock+0x177/0x190 [ 4548.179900] _raw_spin_lock+0x21/0x30 [ 4548.181900] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4548.185813] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4548.189965] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4548.193556] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4548.196900] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4548.200418] ? woken_wake_function+0x30/0x30 [ 4548.202858] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4548.207712] kthread+0x1d1/0x200 [ 4548.209371] ? set_kthread_struct+0x70/0x70 [ 4548.211906] ret_from_fork+0x1f/0x30 [ 4552.056017] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 4552.059081] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4552.063022] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4552.080174] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4552.080186] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4552.080189] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4552.080203] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4552.080207] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4552.080212] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4552.080215] RDX: 0000000000002cf6 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4552.080217] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4552.080220] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4552.080222] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4552.080226] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4552.080229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4552.080231] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4552.080239] Call Trace: [ 4552.080244] [ 4552.080250] ? show_regs.cold.9+0x22/0x2f [ 4552.080259] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4552.080266] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4552.080270] ? watchdog+0x40/0x40 [ 4552.080274] ? hrtimer_interrupt+0x128/0x2c0 [ 4552.080278] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4552.080285] ? apic_timer_interrupt+0xf/0x20 [ 4552.080289] [ 4552.080292] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4552.080296] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4552.080300] do_raw_spin_lock+0x177/0x190 [ 4552.080304] _raw_spin_lock+0x21/0x30 [ 4552.080310] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4552.080697] obd_mod_free+0x46/0x140 [ptlrpc] [ 4552.080966] mdc_commit_open+0x91/0xb0 [mdc] [ 4552.081056] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4552.081279] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4552.081548] after_reply+0xe2c/0x1780 [ptlrpc] [ 4552.081745] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4552.082191] ? __next_timer_interrupt+0x160/0x160 [ 4552.082200] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4552.082484] ? do_wait_intr+0xf0/0xf0 [ 4552.082492] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4552.082682] kthread+0x1d1/0x200 [ 4552.082687] ? set_kthread_struct+0x70/0x70 [ 4552.082691] ret_from_fork+0x1f/0x30 [ 4552.423194] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4552.500874] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4552.509722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4552.514739] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4552.519254] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4552.531653] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4552.537300] RAX: 0000000000007609 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4552.543493] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4552.548050] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4552.557650] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4552.570933] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4552.578545] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4552.583612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4552.587854] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4552.592670] Call Trace: [ 4552.594300] [ 4552.595176] ? show_regs.cold.9+0x22/0x2f [ 4552.597925] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4552.600878] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4552.603669] ? watchdog+0x40/0x40 [ 4552.609031] ? hrtimer_interrupt+0x128/0x2c0 [ 4552.613600] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4552.617919] ? apic_timer_interrupt+0xf/0x20 [ 4552.621754] [ 4552.623194] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4552.626777] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4552.631377] do_raw_spin_lock+0x177/0x190 [ 4552.634089] _raw_spin_lock+0x21/0x30 [ 4552.636386] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4552.640072] ? do_raw_spin_unlock+0x75/0x190 [ 4552.642514] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4552.646002] ? do_wait_intr+0xf0/0xf0 [ 4552.648902] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4552.652456] mdc_reint+0x2d/0x110 [mdc] [ 4552.654668] mdc_unlink+0x658/0x870 [mdc] [ 4552.657666] lmv_unlink+0x34d/0xbd0 [lmv] [ 4552.660529] ll_unlink+0x24d/0x7f0 [lustre] [ 4552.664503] vfs_unlink+0x202/0x330 [ 4552.667561] do_unlinkat+0x2fa/0x460 [ 4552.669926] __x64_sys_unlink+0x37/0x50 [ 4552.672513] do_syscall_64+0xc1/0x440 [ 4552.674749] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4552.678714] RIP: 0033:0x7f83e5f5c10b [ 4552.682128] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4552.695108] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4552.700735] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4552.705296] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4552.711024] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4552.718485] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4552.724401] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4554.214837] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2041s! [ 4554.223013] Showing busy workqueues and worker pools: [ 4554.226611] workqueue events: flags=0x0 [ 4554.232164] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4554.244267] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4554.253609] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4554.260743] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4554.266749] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4554.273427] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4554.285379] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4554.296436] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4554.305048] workqueue events_long: flags=0x0 [ 4554.307969] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4554.321426] pending: lu_objects_flush [obdclass] [ 4554.326358] workqueue events_unbound: flags=0x2 [ 4554.330843] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4554.341400] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4554.343645] workqueue events_power_efficient: flags=0x80 [ 4554.351937] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4554.365320] pending: neigh_periodic_work, do_cache_clean [ 4554.372604] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4554.386331] pending: neigh_periodic_work, check_lifetime [ 4554.401895] workqueue mm_percpu_wq: flags=0x8 [ 4554.406550] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4554.414429] pending: vmstat_update, lru_add_drain_per_cpu [ 4554.419635] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4554.428279] pending: vmstat_update [ 4554.430926] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4554.438194] pending: lru_add_drain_per_cpu BAR(48) [ 4554.451076] workqueue ipv6_addrconf: flags=0x40008 [ 4554.460466] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4554.469259] pending: addrconf_verify_work [ 4554.474827] workqueue knbd0-recv: flags=0x1a [ 4554.485223] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4554.496499] in-flight: 650:recv_work [ 4554.498033] workqueue ptlrpc_pinger: flags=0x2 [ 4554.511165] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4554.522927] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4554.525139] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2041s workers=4 idle: 36 466 29757 [ 4554.543916] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4544s workers=2 idle: 652 [ 4561.015859] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 4561.022828] rcu: 0-....: (1604365 ticks this GP) idle=30a/1/0x4000000000000000 softirq=128519/128519 fqs=126395 [ 4561.029851] (detected by 1, t=2580056 jiffies, g=262441, q=57895) [ 4561.035578] Sending NMI from CPU 1 to CPUs 0: [ 4561.041009] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 4561.041106] rcu: rcu_sched kthread starved for 2049243 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 4561.055846] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4561.059793] rcu: RCU grace-period kthread stack dump: [ 4561.061901] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4561.067809] Call Trace: [ 4561.071262] __schedule+0x351/0xcb0 [ 4561.074878] schedule+0xc0/0x180 [ 4561.076081] schedule_timeout+0xb4/0x190 [ 4561.079327] ? do_raw_spin_unlock+0x75/0x190 [ 4561.089561] ? __next_timer_interrupt+0x160/0x160 [ 4561.098891] ? prepare_to_swait_event+0xf3/0x140 [ 4561.105359] rcu_gp_kthread+0x8dd/0x15b0 [ 4561.107918] ? force_qs_rnp+0x2c0/0x2c0 [ 4561.115766] kthread+0x1d1/0x200 [ 4561.117857] ? set_kthread_struct+0x70/0x70 [ 4561.119559] ret_from_fork+0x1f/0x30 [ 4561.120821] rcu: Stack dump where RCU GP kthread last ran: [ 4561.123115] Sending NMI from CPU 1 to CPUs 2: [ 4561.126887] NMI backtrace for cpu 2 [ 4561.126939] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4561.126942] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4561.126943] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4561.126965] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4561.126967] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4561.126969] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 [ 4561.126971] RAX: 0000000000007584 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4561.126973] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4561.126978] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4561.126980] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4561.126981] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4561.126983] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4561.126985] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4561.126986] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4561.126987] Call Trace: [ 4561.126988] [ 4561.126989] ? show_regs.cold.9+0x22/0x2f [ 4561.126990] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4561.126991] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4561.126992] ? nmi_handle+0xbd/0x210 [ 4561.126993] ? default_do_nmi+0x4b/0x1b0 [ 4561.126994] ? do_nmi+0x280/0x380 [ 4561.126995] ? end_repeat_nmi+0x16/0x1f [ 4561.127011] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4561.127015] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4561.127016] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4561.127018] [ 4561.127019] do_raw_spin_lock+0x177/0x190 [ 4561.127019] _raw_spin_lock+0x21/0x30 [ 4561.127020] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4561.127021] process_one_work+0x2c8/0x700 [ 4561.127022] worker_thread+0x296/0x6e0 [ 4561.127023] ? rescuer_thread+0x570/0x570 [ 4561.127024] kthread+0x1d1/0x200 [ 4561.127025] ? set_kthread_struct+0x70/0x70 [ 4561.127026] ret_from_fork+0x1f/0x30 [ 4580.062358] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4580.064763] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4580.087883] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4580.094138] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4580.098998] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4580.102349] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4580.112463] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4580.116262] RAX: 000000000000620f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4580.122244] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4580.125624] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4580.133096] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4580.137049] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4580.141003] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4580.144778] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4580.147898] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4580.151301] Call Trace: [ 4580.152430] [ 4580.153258] ? show_regs.cold.9+0x22/0x2f [ 4580.155625] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4580.157924] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4580.160049] ? watchdog+0x40/0x40 [ 4580.162106] ? hrtimer_interrupt+0x128/0x2c0 [ 4580.163830] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4580.168630] ? apic_timer_interrupt+0xf/0x20 [ 4580.170544] [ 4580.172084] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4580.174417] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4580.176433] do_raw_spin_lock+0x177/0x190 [ 4580.178414] _raw_spin_lock+0x21/0x30 [ 4580.180250] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4580.183019] ? do_raw_spin_unlock+0x75/0x190 [ 4580.184826] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4580.187101] ? do_wait_intr+0xf0/0xf0 [ 4580.188648] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4580.190895] mdc_reint+0x2d/0x110 [mdc] [ 4580.192634] mdc_unlink+0x658/0x870 [mdc] [ 4580.194362] lmv_unlink+0x34d/0xbd0 [lmv] [ 4580.196335] ll_unlink+0x24d/0x7f0 [lustre] [ 4580.198102] vfs_unlink+0x202/0x330 [ 4580.199949] do_unlinkat+0x2fa/0x460 [ 4580.202153] __x64_sys_unlink+0x37/0x50 [ 4580.204107] do_syscall_64+0xc1/0x440 [ 4580.205977] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4580.208392] RIP: 0033:0x7f83e5f5c10b [ 4580.210083] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4580.220064] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4580.223969] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4580.227336] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4580.231268] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4580.235086] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4580.238125] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4584.938591] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2071s! [ 4584.942543] Showing busy workqueues and worker pools: [ 4584.945488] workqueue events: flags=0x0 [ 4584.947686] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4584.951568] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4584.959439] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4584.966642] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4584.971064] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4584.980448] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4584.994307] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4585.000276] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4585.006136] workqueue events_long: flags=0x0 [ 4585.009451] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4585.017729] pending: lu_objects_flush [obdclass] [ 4585.021302] workqueue events_unbound: flags=0x2 [ 4585.023870] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4585.027089] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4585.030125] workqueue events_power_efficient: flags=0x80 [ 4585.032539] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4585.035386] pending: neigh_periodic_work, do_cache_clean [ 4585.037525] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4585.040210] pending: neigh_periodic_work, check_lifetime [ 4585.042748] workqueue mm_percpu_wq: flags=0x8 [ 4585.044546] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4585.047444] pending: vmstat_update, lru_add_drain_per_cpu [ 4585.049563] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4585.052614] pending: vmstat_update [ 4585.054153] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4585.062120] pending: lru_add_drain_per_cpu BAR(48) [ 4585.067111] workqueue ipv6_addrconf: flags=0x40008 [ 4585.074158] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4585.081879] pending: addrconf_verify_work [ 4585.086245] workqueue knbd0-recv: flags=0x1a [ 4585.097327] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4585.109574] in-flight: 650:recv_work [ 4585.117364] workqueue ptlrpc_pinger: flags=0x2 [ 4585.126339] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4585.142583] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4585.149101] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2071s workers=4 idle: 36 466 29757 [ 4585.155413] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4575s workers=2 idle: 652 [ 4588.061959] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 4588.067167] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 4588.068907] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4588.068984] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4588.068991] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4588.068995] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4588.069010] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4588.069014] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4588.069019] RAX: 0000000000000d86 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4588.069021] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4588.069833] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4588.069836] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4588.069839] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4588.069842] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4588.069845] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4588.069847] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4588.069856] Call Trace: [ 4588.069861] [ 4588.069867] ? show_regs.cold.9+0x22/0x2f [ 4588.069877] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4588.069883] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4588.069888] ? watchdog+0x40/0x40 [ 4588.069893] ? hrtimer_interrupt+0x128/0x2c0 [ 4588.069897] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4588.069906] ? apic_timer_interrupt+0xf/0x20 [ 4588.069911] [ 4588.069914] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4588.069919] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4588.069925] do_raw_spin_lock+0x177/0x190 [ 4588.069930] _raw_spin_lock+0x21/0x30 [ 4588.069935] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4588.071627] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4588.072711] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4588.073012] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4588.073987] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4588.075505] ? woken_wake_function+0x30/0x30 [ 4588.075517] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4588.075849] kthread+0x1d1/0x200 [ 4588.075856] ? set_kthread_struct+0x70/0x70 [ 4588.075861] ret_from_fork+0x1f/0x30 [ 4588.366102] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4588.400092] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4588.405172] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4588.414387] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4588.425861] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4588.433330] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4588.453655] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4588.463934] RAX: 0000000000004288 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4588.477274] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4588.484817] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4588.493652] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4588.501633] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4588.510081] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4588.518068] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4588.525195] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4588.531143] Call Trace: [ 4588.533053] [ 4588.534860] ? show_regs.cold.9+0x22/0x2f [ 4588.541119] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4588.548055] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4588.550923] ? watchdog+0x40/0x40 [ 4588.556307] ? hrtimer_interrupt+0x128/0x2c0 [ 4588.559337] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4588.568228] ? apic_timer_interrupt+0xf/0x20 [ 4588.574677] [ 4588.581066] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4588.592045] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4588.599552] do_raw_spin_lock+0x177/0x190 [ 4588.601967] _raw_spin_lock+0x21/0x30 [ 4588.608389] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4588.616608] process_one_work+0x2c8/0x700 [ 4588.618474] worker_thread+0x296/0x6e0 [ 4588.623231] ? rescuer_thread+0x570/0x570 [ 4588.626292] kthread+0x1d1/0x200 [ 4588.630789] ? set_kthread_struct+0x70/0x70 [ 4588.633372] ret_from_fork+0x1f/0x30 [ 4608.062019] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4608.064929] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4608.144991] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4608.153378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4608.159772] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4608.167755] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4608.196374] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4608.207558] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4608.214896] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4608.220923] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4608.227689] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4608.231794] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4608.235043] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4608.239077] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4608.241378] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4608.244922] Call Trace: [ 4608.246119] [ 4608.247147] ? show_regs.cold.9+0x22/0x2f [ 4608.249037] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4608.251180] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4608.253540] ? watchdog+0x40/0x40 [ 4608.254913] ? hrtimer_interrupt+0x128/0x2c0 [ 4608.256972] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4608.259112] ? apic_timer_interrupt+0xf/0x20 [ 4608.260879] [ 4608.261733] ? native_safe_halt+0x2e/0x50 [ 4608.263464] kvm_wait+0xcd/0xe0 [ 4608.264720] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4608.266850] do_raw_spin_lock+0x177/0x190 [ 4608.268531] _raw_spin_lock+0x21/0x30 [ 4608.269964] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4608.272397] ? do_raw_spin_unlock+0x75/0x190 [ 4608.274120] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4608.276357] ? do_wait_intr+0xf0/0xf0 [ 4608.277850] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4608.280698] mdc_reint+0x2d/0x110 [mdc] [ 4608.282978] mdc_unlink+0x658/0x870 [mdc] [ 4608.285823] lmv_unlink+0x34d/0xbd0 [lmv] [ 4608.288649] ll_unlink+0x24d/0x7f0 [lustre] [ 4608.291566] vfs_unlink+0x202/0x330 [ 4608.296327] do_unlinkat+0x2fa/0x460 [ 4608.298593] __x64_sys_unlink+0x37/0x50 [ 4608.302155] do_syscall_64+0xc1/0x440 [ 4608.305090] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4608.310057] RIP: 0033:0x7f83e5f5c10b [ 4608.314308] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4608.323056] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4608.328158] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4608.331649] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4608.335717] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4608.339319] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4608.344764] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4612.061934] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 4612.071793] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4612.088911] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4612.094112] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4612.098013] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4612.100914] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4612.109830] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4612.113539] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4612.116865] RDX: 0000000000007ba5 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4612.120686] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4612.123490] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4612.126727] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4612.129998] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4612.133819] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4612.136351] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4612.139877] Call Trace: [ 4612.141735] [ 4612.142769] ? show_regs.cold.9+0x22/0x2f [ 4612.144804] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4612.147494] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4612.149768] ? watchdog+0x40/0x40 [ 4612.151764] ? hrtimer_interrupt+0x128/0x2c0 [ 4612.153977] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4612.156538] ? apic_timer_interrupt+0xf/0x20 [ 4612.158760] [ 4612.159785] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4612.162391] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4612.165067] do_raw_spin_lock+0x177/0x190 [ 4612.167390] _raw_spin_lock+0x21/0x30 [ 4612.169400] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4612.174186] obd_mod_free+0x46/0x140 [ptlrpc] [ 4612.177346] mdc_commit_open+0x91/0xb0 [mdc] [ 4612.179604] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4612.182937] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4612.186502] after_reply+0xe2c/0x1780 [ptlrpc] [ 4612.189236] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4612.192565] ? __next_timer_interrupt+0x160/0x160 [ 4612.194917] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4612.197540] ? do_wait_intr+0xf0/0xf0 [ 4612.199451] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4612.202767] kthread+0x1d1/0x200 [ 4612.203928] ? set_kthread_struct+0x70/0x70 [ 4612.205938] ret_from_fork+0x1f/0x30 [ 4615.650468] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2102s! [ 4615.654554] Showing busy workqueues and worker pools: [ 4615.657018] workqueue events: flags=0x0 [ 4615.661928] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4615.665548] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4615.670143] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4615.673439] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4615.679514] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4615.683932] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4615.689220] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4615.693482] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4615.699365] workqueue events_long: flags=0x0 [ 4615.702504] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4615.710068] pending: lu_objects_flush [obdclass] [ 4615.713641] workqueue events_unbound: flags=0x2 [ 4615.716357] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4615.726621] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4615.730021] workqueue events_power_efficient: flags=0x80 [ 4615.741021] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4615.750235] pending: neigh_periodic_work, do_cache_clean [ 4615.761140] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4615.769541] pending: neigh_periodic_work, check_lifetime [ 4615.776169] workqueue mm_percpu_wq: flags=0x8 [ 4615.779583] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4615.784576] pending: vmstat_update, lru_add_drain_per_cpu [ 4615.791947] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4615.796093] pending: vmstat_update [ 4615.798221] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4615.803468] pending: lru_add_drain_per_cpu BAR(48) [ 4615.808026] workqueue ipv6_addrconf: flags=0x40008 [ 4615.811087] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4615.821528] pending: addrconf_verify_work [ 4615.825609] workqueue knbd0-recv: flags=0x1a [ 4615.829803] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4615.832847] in-flight: 650:recv_work [ 4615.835027] workqueue ptlrpc_pinger: flags=0x2 [ 4615.841021] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4615.843789] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4615.846863] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2102s workers=4 idle: 36 466 29757 [ 4615.851327] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4605s workers=2 idle: 652 [ 4616.060553] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 4616.062876] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 4616.064329] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4616.064461] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4616.064468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4616.064475] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4616.064993] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4616.065005] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4616.065009] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4616.065014] RAX: 0000000000002753 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4616.065016] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4616.065019] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4616.065021] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4616.065024] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4616.065028] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4616.065031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4616.065033] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4616.065043] Call Trace: [ 4616.065048] [ 4616.065054] ? show_regs.cold.9+0x22/0x2f [ 4616.065064] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4616.065070] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4616.065075] ? watchdog+0x40/0x40 [ 4616.065080] ? hrtimer_interrupt+0x128/0x2c0 [ 4616.065084] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4616.065092] ? apic_timer_interrupt+0xf/0x20 [ 4616.065097] [ 4616.065100] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4616.065105] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4616.065110] do_raw_spin_lock+0x177/0x190 [ 4616.065116] _raw_spin_lock+0x21/0x30 [ 4616.065120] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4616.066516] process_one_work+0x2c8/0x700 [ 4616.066526] worker_thread+0x296/0x6e0 [ 4616.066530] ? rescuer_thread+0x570/0x570 [ 4616.066535] kthread+0x1d1/0x200 [ 4616.066540] ? set_kthread_struct+0x70/0x70 [ 4616.066544] ret_from_fork+0x1f/0x30 [ 4616.376233] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4616.439200] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4616.452612] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4616.457355] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4616.467805] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4616.487526] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4616.495265] RAX: 0000000000000f90 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4616.503862] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4616.508814] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4616.515094] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4616.521427] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4616.524429] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4616.527352] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4616.534836] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4616.537974] Call Trace: [ 4616.539024] [ 4616.540964] ? show_regs.cold.9+0x22/0x2f [ 4616.544104] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4616.545799] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4616.548054] ? watchdog+0x40/0x40 [ 4616.550978] ? hrtimer_interrupt+0x128/0x2c0 [ 4616.553009] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4616.554322] ? apic_timer_interrupt+0xf/0x20 [ 4616.557778] [ 4616.558585] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4616.562946] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4616.565930] do_raw_spin_lock+0x177/0x190 [ 4616.568047] _raw_spin_lock+0x21/0x30 [ 4616.570359] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4616.578098] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4616.581213] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4616.586938] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4616.590088] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4616.594828] ? woken_wake_function+0x30/0x30 [ 4616.596825] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4616.599823] kthread+0x1d1/0x200 [ 4616.601511] ? set_kthread_struct+0x70/0x70 [ 4616.606329] ret_from_fork+0x1f/0x30 [ 4636.066323] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4636.071312] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4636.101298] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4636.111879] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4636.123700] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4636.127789] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4636.144772] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4636.149954] RAX: 0000000000002387 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4636.155496] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4636.161478] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4636.166288] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4636.173492] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4636.181775] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4636.189284] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4636.193467] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4636.196488] Call Trace: [ 4636.198123] [ 4636.207883] ? show_regs.cold.9+0x22/0x2f [ 4636.209763] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4636.211595] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4636.218758] ? watchdog+0x40/0x40 [ 4636.221095] ? hrtimer_interrupt+0x128/0x2c0 [ 4636.231405] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4636.241363] ? apic_timer_interrupt+0xf/0x20 [ 4636.246671] [ 4636.248454] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4636.258481] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4636.265788] do_raw_spin_lock+0x177/0x190 [ 4636.267988] _raw_spin_lock+0x21/0x30 [ 4636.272841] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4636.276787] ? do_raw_spin_unlock+0x75/0x190 [ 4636.279368] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4636.284312] ? do_wait_intr+0xf0/0xf0 [ 4636.285695] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4636.294578] mdc_reint+0x2d/0x110 [mdc] [ 4636.298815] mdc_unlink+0x658/0x870 [mdc] [ 4636.302859] lmv_unlink+0x34d/0xbd0 [lmv] [ 4636.306996] ll_unlink+0x24d/0x7f0 [lustre] [ 4636.313758] vfs_unlink+0x202/0x330 [ 4636.315336] do_unlinkat+0x2fa/0x460 [ 4636.321419] __x64_sys_unlink+0x37/0x50 [ 4636.322837] do_syscall_64+0xc1/0x440 [ 4636.331426] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4636.335027] RIP: 0033:0x7f83e5f5c10b [ 4636.337071] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4636.349068] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4636.353746] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4636.358430] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4636.361933] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4636.366845] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4636.370243] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4640.056112] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 4640.061153] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4640.090006] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4640.100844] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4640.106465] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4640.109513] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4640.121099] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4640.125301] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4640.127919] RDX: 0000000000004e02 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4640.131402] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4640.134945] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4640.138226] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4640.141505] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4640.144506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4640.146974] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4640.149269] Call Trace: [ 4640.150065] [ 4640.150668] ? show_regs.cold.9+0x22/0x2f [ 4640.152406] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4640.154361] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4640.156363] ? watchdog+0x40/0x40 [ 4640.157388] ? hrtimer_interrupt+0x128/0x2c0 [ 4640.158630] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4640.161491] ? apic_timer_interrupt+0xf/0x20 [ 4640.165560] [ 4640.167436] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4640.174495] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4640.183470] do_raw_spin_lock+0x177/0x190 [ 4640.185407] _raw_spin_lock+0x21/0x30 [ 4640.187427] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4640.192606] obd_mod_free+0x46/0x140 [ptlrpc] [ 4640.195611] mdc_commit_open+0x91/0xb0 [mdc] [ 4640.198410] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4640.202035] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4640.210564] after_reply+0xe2c/0x1780 [ptlrpc] [ 4640.212566] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4640.216354] ? __next_timer_interrupt+0x160/0x160 [ 4640.220136] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4640.223512] ? do_wait_intr+0xf0/0xf0 [ 4640.224965] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4640.228210] kthread+0x1d1/0x200 [ 4640.230406] ? set_kthread_struct+0x70/0x70 [ 4640.232419] ret_from_fork+0x1f/0x30 [ 4644.059025] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 4644.060054] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 4644.063460] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4644.063606] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4644.063613] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4644.063616] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4644.063629] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4644.063634] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4644.063638] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4644.063640] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4644.063643] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4644.063645] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4644.063648] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4644.063651] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4644.063654] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4644.063657] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4644.063665] Call Trace: [ 4644.063670] [ 4644.063677] ? show_regs.cold.9+0x22/0x2f [ 4644.063685] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4644.063692] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4644.063696] ? watchdog+0x40/0x40 [ 4644.063700] ? hrtimer_interrupt+0x128/0x2c0 [ 4644.063704] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4644.063710] ? apic_timer_interrupt+0xf/0x20 [ 4644.063715] [ 4644.063718] ? native_safe_halt+0x2e/0x50 [ 4644.063722] kvm_wait+0xcd/0xe0 [ 4644.063729] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4644.063738] do_raw_spin_lock+0x177/0x190 [ 4644.063742] _raw_spin_lock+0x21/0x30 [ 4644.063746] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4644.064082] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4644.067381] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4644.067459] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4644.067466] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4644.067472] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4644.067743] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4644.067752] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4644.068015] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4644.068021] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4644.068024] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4644.068026] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4644.068028] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4644.068031] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4644.068034] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4644.068037] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4644.068039] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4644.068048] Call Trace: [ 4644.068054] [ 4644.068059] ? show_regs.cold.9+0x22/0x2f [ 4644.068071] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4644.068077] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4644.068081] ? watchdog+0x40/0x40 [ 4644.068085] ? hrtimer_interrupt+0x128/0x2c0 [ 4644.068090] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4644.068097] ? apic_timer_interrupt+0xf/0x20 [ 4644.068169] [ 4644.068174] ? native_safe_halt+0x2e/0x50 [ 4644.068186] kvm_wait+0xcd/0xe0 [ 4644.068195] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4644.068203] do_raw_spin_lock+0x177/0x190 [ 4644.068209] _raw_spin_lock+0x21/0x30 [ 4644.068213] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4644.068438] process_one_work+0x2c8/0x700 [ 4644.068445] worker_thread+0x296/0x6e0 [ 4644.068450] ? rescuer_thread+0x570/0x570 [ 4644.068454] kthread+0x1d1/0x200 [ 4644.068459] ? set_kthread_struct+0x70/0x70 [ 4644.068463] ret_from_fork+0x1f/0x30 [ 4644.369005] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4644.375443] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4644.381782] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4644.387391] ? woken_wake_function+0x30/0x30 [ 4644.391748] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4644.398786] kthread+0x1d1/0x200 [ 4644.400846] ? set_kthread_struct+0x70/0x70 [ 4644.407337] ret_from_fork+0x1f/0x30 [ 4645.860464] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2132s! [ 4645.866967] Showing busy workqueues and worker pools: [ 4645.869622] workqueue events: flags=0x0 [ 4645.871572] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4645.874539] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4645.878465] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4645.880655] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4645.882877] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4645.886204] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4645.889900] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4645.894283] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4645.898641] workqueue events_long: flags=0x0 [ 4645.900500] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4645.904461] pending: lu_objects_flush [obdclass] [ 4645.908659] workqueue events_unbound: flags=0x2 [ 4645.910686] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4645.913929] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4645.916668] workqueue events_power_efficient: flags=0x80 [ 4645.919493] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4645.922068] pending: neigh_periodic_work, do_cache_clean [ 4645.925656] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4645.928975] pending: neigh_periodic_work, check_lifetime [ 4645.931947] workqueue mm_percpu_wq: flags=0x8 [ 4645.933927] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4645.936870] pending: vmstat_update, lru_add_drain_per_cpu [ 4645.939654] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4645.942562] pending: vmstat_update [ 4645.944752] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4645.948334] pending: lru_add_drain_per_cpu BAR(48) [ 4645.950661] workqueue ipv6_addrconf: flags=0x40008 [ 4645.953171] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4645.956232] pending: addrconf_verify_work [ 4645.958331] workqueue knbd0-recv: flags=0x1a [ 4645.960624] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4645.963937] in-flight: 650:recv_work [ 4645.965662] workqueue ptlrpc_pinger: flags=0x2 [ 4645.967686] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4645.972576] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4645.979070] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2132s workers=4 idle: 36 466 29757 [ 4645.984118] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4636s workers=2 idle: 652 [ 4664.062086] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4664.066269] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4664.087644] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4664.092825] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4664.097185] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4664.100131] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4664.108515] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4664.112367] RAX: 0000000000006f14 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4664.115647] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4664.119230] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4664.122554] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4664.125280] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4664.128568] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4664.132408] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4664.135106] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4664.138299] Call Trace: [ 4664.139251] [ 4664.139938] ? show_regs.cold.9+0x22/0x2f [ 4664.141791] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4664.144190] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4664.146182] ? watchdog+0x40/0x40 [ 4664.147615] ? hrtimer_interrupt+0x128/0x2c0 [ 4664.149554] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4664.151676] ? apic_timer_interrupt+0xf/0x20 [ 4664.153957] [ 4664.154965] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4664.157492] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4664.160338] do_raw_spin_lock+0x177/0x190 [ 4664.162200] _raw_spin_lock+0x21/0x30 [ 4664.164089] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4664.166608] ? do_raw_spin_unlock+0x75/0x190 [ 4664.169110] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4664.171440] ? do_wait_intr+0xf0/0xf0 [ 4664.173162] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4664.175582] mdc_reint+0x2d/0x110 [mdc] [ 4664.177367] mdc_unlink+0x658/0x870 [mdc] [ 4664.180829] lmv_unlink+0x34d/0xbd0 [lmv] [ 4664.182952] ll_unlink+0x24d/0x7f0 [lustre] [ 4664.192000] vfs_unlink+0x202/0x330 [ 4664.193364] do_unlinkat+0x2fa/0x460 [ 4664.194665] __x64_sys_unlink+0x37/0x50 [ 4664.198464] do_syscall_64+0xc1/0x440 [ 4664.199778] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4664.202456] RIP: 0033:0x7f83e5f5c10b [ 4664.203731] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4664.211217] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4664.214956] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4664.218629] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4664.221895] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4664.224561] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4664.227384] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4672.057086] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 4672.060048] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 4672.060441] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 4672.060448] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4672.060526] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4672.060532] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4672.060535] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4672.060549] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4672.060553] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4672.060558] RAX: 0000000000007e9a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4672.060560] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4672.060562] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4672.060564] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4672.060567] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4672.060571] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4672.060573] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4672.060575] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4672.060585] Call Trace: [ 4672.060591] [ 4672.060597] ? show_regs.cold.9+0x22/0x2f [ 4672.060607] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4672.060616] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4672.060620] ? watchdog+0x40/0x40 [ 4672.060624] ? hrtimer_interrupt+0x128/0x2c0 [ 4672.060645] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4672.060656] ? apic_timer_interrupt+0xf/0x20 [ 4672.060661] [ 4672.060663] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4672.060669] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4672.060674] do_raw_spin_lock+0x177/0x190 [ 4672.060679] _raw_spin_lock+0x21/0x30 [ 4672.060684] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4672.060956] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4672.062178] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4672.062866] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4672.063545] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4672.063776] ? woken_wake_function+0x30/0x30 [ 4672.063786] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4672.063996] kthread+0x1d1/0x200 [ 4672.064003] ? set_kthread_struct+0x70/0x70 [ 4672.064799] ret_from_fork+0x1f/0x30 [ 4672.066354] Modules linked in: lustre(O) [ 4672.068399] Modules linked in: lustre(O) [ 4672.072573] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4672.072708] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4672.072715] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4672.072718] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4672.072733] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4672.072737] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4672.072741] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4672.072743] RDX: 000000000000127e RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4672.072746] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4672.072748] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4672.072750] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4672.072754] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4672.072757] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4672.072759] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4672.072767] Call Trace: [ 4672.072772] [ 4672.072776] ? show_regs.cold.9+0x22/0x2f [ 4672.072785] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4672.072791] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4672.072794] ? watchdog+0x40/0x40 [ 4672.072798] ? hrtimer_interrupt+0x128/0x2c0 [ 4672.072802] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4672.072810] ? apic_timer_interrupt+0xf/0x20 [ 4672.072814] [ 4672.072816] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4672.072820] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4672.072824] do_raw_spin_lock+0x177/0x190 [ 4672.072880] _raw_spin_lock+0x21/0x30 [ 4672.072885] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4672.073197] obd_mod_free+0x46/0x140 [ptlrpc] [ 4672.073371] mdc_commit_open+0x91/0xb0 [mdc] [ 4672.073407] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4672.073599] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4672.087804] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4672.087868] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4672.087873] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4672.087878] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4672.088199] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4672.088272] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4672.088274] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4672.088278] RAX: 000000000000101e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4672.088280] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4672.088281] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4672.088283] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4672.088284] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4672.088287] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4672.088289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4672.088291] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4672.088297] Call Trace: [ 4672.088301] [ 4672.088305] ? show_regs.cold.9+0x22/0x2f [ 4672.088316] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4672.088322] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4672.088326] ? watchdog+0x40/0x40 [ 4672.088331] ? hrtimer_interrupt+0x128/0x2c0 [ 4672.088335] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4672.088343] ? apic_timer_interrupt+0xf/0x20 [ 4672.088347] [ 4672.088350] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4672.088354] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4672.088359] do_raw_spin_lock+0x177/0x190 [ 4672.088364] _raw_spin_lock+0x21/0x30 [ 4672.088368] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4672.088586] process_one_work+0x2c8/0x700 [ 4672.088592] worker_thread+0x296/0x6e0 [ 4672.088600] ? rescuer_thread+0x570/0x570 [ 4672.088603] kthread+0x1d1/0x200 [ 4672.088607] ? set_kthread_struct+0x70/0x70 [ 4672.088610] ret_from_fork+0x1f/0x30 [ 4672.450055] after_reply+0xe2c/0x1780 [ptlrpc] [ 4672.452795] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4672.455502] ? __next_timer_interrupt+0x160/0x160 [ 4672.458373] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4672.460792] ? do_wait_intr+0xf0/0xf0 [ 4672.462963] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4672.466191] kthread+0x1d1/0x200 [ 4672.467910] ? set_kthread_struct+0x70/0x70 [ 4672.469953] ret_from_fork+0x1f/0x30 [ 4676.069552] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2162s! [ 4676.073511] Showing busy workqueues and worker pools: [ 4676.076217] workqueue events: flags=0x0 [ 4676.077936] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4676.082557] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4676.088578] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4676.093081] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4676.096431] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4676.099398] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4676.103417] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4676.106495] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4676.111699] workqueue events_long: flags=0x0 [ 4676.113507] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4676.117778] pending: lu_objects_flush [obdclass] [ 4676.120244] workqueue events_unbound: flags=0x2 [ 4676.122223] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4676.125706] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4676.128189] workqueue events_power_efficient: flags=0x80 [ 4676.130372] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4676.133144] pending: neigh_periodic_work, do_cache_clean [ 4676.135449] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4676.138496] pending: neigh_periodic_work, check_lifetime [ 4676.140844] workqueue mm_percpu_wq: flags=0x8 [ 4676.142459] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4676.145341] pending: vmstat_update, lru_add_drain_per_cpu [ 4676.147716] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4676.150423] pending: vmstat_update [ 4676.152286] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4676.155312] pending: lru_add_drain_per_cpu BAR(48) [ 4676.157412] workqueue ipv6_addrconf: flags=0x40008 [ 4676.159352] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4676.162207] pending: addrconf_verify_work [ 4676.164096] workqueue knbd0-recv: flags=0x1a [ 4676.165955] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4676.169060] in-flight: 650:recv_work [ 4676.171127] workqueue ptlrpc_pinger: flags=0x2 [ 4676.173396] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4676.176356] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4676.179485] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2162s workers=4 idle: 36 466 29757 [ 4676.183482] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4666s workers=2 idle: 652 [ 4692.062404] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4692.072496] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4692.115949] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4692.121149] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4692.125015] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4692.127456] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4692.135634] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4692.139248] RAX: 0000000000005e0a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4692.143016] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4692.147161] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4692.150221] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4692.160624] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4692.165613] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4692.174328] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4692.184654] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4692.196882] Call Trace: [ 4692.199535] [ 4692.202384] ? show_regs.cold.9+0x22/0x2f [ 4692.208176] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4692.211957] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4692.214518] ? watchdog+0x40/0x40 [ 4692.216619] ? hrtimer_interrupt+0x128/0x2c0 [ 4692.219610] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4692.222740] ? apic_timer_interrupt+0xf/0x20 [ 4692.225209] [ 4692.231580] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4692.233404] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4692.238053] do_raw_spin_lock+0x177/0x190 [ 4692.240089] _raw_spin_lock+0x21/0x30 [ 4692.243739] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4692.248243] ? do_raw_spin_unlock+0x75/0x190 [ 4692.251985] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4692.254382] ? do_wait_intr+0xf0/0xf0 [ 4692.259077] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4692.261339] mdc_reint+0x2d/0x110 [mdc] [ 4692.264823] mdc_unlink+0x658/0x870 [mdc] [ 4692.267346] lmv_unlink+0x34d/0xbd0 [lmv] [ 4692.269592] ll_unlink+0x24d/0x7f0 [lustre] [ 4692.273613] vfs_unlink+0x202/0x330 [ 4692.275734] do_unlinkat+0x2fa/0x460 [ 4692.278236] __x64_sys_unlink+0x37/0x50 [ 4692.279989] do_syscall_64+0xc1/0x440 [ 4692.281696] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4692.283187] RIP: 0033:0x7f83e5f5c10b [ 4692.286321] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4692.302084] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4692.308378] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4692.311932] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4692.316066] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4692.323474] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4692.330645] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4700.060616] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 4700.064522] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 4700.066317] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4700.066401] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4700.066407] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4700.066413] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4700.066723] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4700.066733] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4700.066737] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4700.066741] RAX: 0000000000003f84 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4700.066744] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4700.066746] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4700.066748] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4700.066750] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4700.066753] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4700.066756] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4700.066758] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4700.066765] Call Trace: [ 4700.066770] [ 4700.066775] ? show_regs.cold.9+0x22/0x2f [ 4700.066783] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4700.066789] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4700.066792] ? watchdog+0x40/0x40 [ 4700.066796] ? hrtimer_interrupt+0x128/0x2c0 [ 4700.066799] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4700.066807] ? apic_timer_interrupt+0xf/0x20 [ 4700.066810] [ 4700.066813] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4700.066816] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4700.066821] do_raw_spin_lock+0x177/0x190 [ 4700.066824] _raw_spin_lock+0x21/0x30 [ 4700.066828] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4700.067045] process_one_work+0x2c8/0x700 [ 4700.067052] worker_thread+0x296/0x6e0 [ 4700.067055] ? rescuer_thread+0x570/0x570 [ 4700.067058] kthread+0x1d1/0x200 [ 4700.067062] ? set_kthread_struct+0x70/0x70 [ 4700.067065] ret_from_fork+0x1f/0x30 [ 4700.305813] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4700.329434] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4700.335118] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4700.340443] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4700.343333] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4700.354017] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4700.358467] RAX: 0000000000006013 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4700.362825] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4700.366002] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4700.369701] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4700.373729] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4700.377012] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4700.380531] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4700.382982] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4700.387216] Call Trace: [ 4700.388477] [ 4700.389315] ? show_regs.cold.9+0x22/0x2f [ 4700.391631] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4700.393955] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4700.396077] ? watchdog+0x40/0x40 [ 4700.397762] ? hrtimer_interrupt+0x128/0x2c0 [ 4700.399818] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4700.401955] ? apic_timer_interrupt+0xf/0x20 [ 4700.403743] [ 4700.404701] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4700.407478] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4700.409535] do_raw_spin_lock+0x177/0x190 [ 4700.411161] _raw_spin_lock+0x21/0x30 [ 4700.413275] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4700.416008] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4700.418139] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4700.421395] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4700.424705] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4700.427611] ? woken_wake_function+0x30/0x30 [ 4700.429381] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4700.432511] kthread+0x1d1/0x200 [ 4700.433543] ? set_kthread_struct+0x70/0x70 [ 4700.435971] ret_from_fork+0x1f/0x30 [ 4704.057081] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 4704.074494] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4704.106205] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4704.111579] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4704.115625] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4704.119574] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4704.131085] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4704.140584] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4704.146503] RDX: 0000000000004354 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4704.151522] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4704.166672] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4704.175292] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4704.182777] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4704.189791] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4704.194664] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4704.199090] Call Trace: [ 4704.200081] [ 4704.200943] ? show_regs.cold.9+0x22/0x2f [ 4704.203253] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4704.206744] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4704.209659] ? watchdog+0x40/0x40 [ 4704.211898] ? hrtimer_interrupt+0x128/0x2c0 [ 4704.214220] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4704.216666] ? apic_timer_interrupt+0xf/0x20 [ 4704.219055] [ 4704.220213] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4704.222834] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4704.226082] do_raw_spin_lock+0x177/0x190 [ 4704.228671] _raw_spin_lock+0x21/0x30 [ 4704.231044] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4704.234285] obd_mod_free+0x46/0x140 [ptlrpc] [ 4704.236947] mdc_commit_open+0x91/0xb0 [mdc] [ 4704.239101] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4704.242541] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4704.246094] after_reply+0xe2c/0x1780 [ptlrpc] [ 4704.253093] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4704.257830] ? __next_timer_interrupt+0x160/0x160 [ 4704.260326] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4704.262445] ? do_wait_intr+0xf0/0xf0 [ 4704.264330] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4704.267438] kthread+0x1d1/0x200 [ 4704.269142] ? set_kthread_struct+0x70/0x70 [ 4704.271310] ret_from_fork+0x1f/0x30 [ 4706.279040] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2193s! [ 4706.288780] Showing busy workqueues and worker pools: [ 4706.293925] workqueue events: flags=0x0 [ 4706.298118] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4706.307143] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4706.317117] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4706.326918] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4706.338094] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4706.346945] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4706.358841] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4706.372192] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4706.377916] workqueue events_long: flags=0x0 [ 4706.379508] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4706.391942] pending: lu_objects_flush [obdclass] [ 4706.402316] workqueue events_unbound: flags=0x2 [ 4706.405797] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4706.411441] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4706.414049] workqueue events_power_efficient: flags=0x80 [ 4706.419339] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4706.422769] pending: neigh_periodic_work, do_cache_clean [ 4706.427316] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4706.433558] pending: neigh_periodic_work, check_lifetime [ 4706.436751] workqueue mm_percpu_wq: flags=0x8 [ 4706.439179] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4706.446932] pending: vmstat_update, lru_add_drain_per_cpu [ 4706.452782] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4706.461225] pending: vmstat_update [ 4706.465165] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4706.472480] pending: lru_add_drain_per_cpu BAR(48) [ 4706.479633] workqueue ipv6_addrconf: flags=0x40008 [ 4706.485650] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4706.489372] pending: addrconf_verify_work [ 4706.492852] workqueue knbd0-recv: flags=0x1a [ 4706.497938] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4706.501759] in-flight: 650:recv_work [ 4706.504383] workqueue ptlrpc_pinger: flags=0x2 [ 4706.507205] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4706.509774] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4706.512977] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2193s workers=4 idle: 36 466 29757 [ 4706.517652] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4696s workers=2 idle: 652 [ 4720.062083] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4720.074597] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4720.140663] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4720.152985] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4720.157475] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4720.162564] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4720.174891] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4720.181784] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4720.185550] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4720.196053] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4720.203641] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4720.215002] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4720.225385] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4720.233533] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4720.245834] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4720.253072] Call Trace: [ 4720.253758] [ 4720.260646] ? show_regs.cold.9+0x22/0x2f [ 4720.263272] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4720.264898] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4720.272626] ? watchdog+0x40/0x40 [ 4720.273614] ? hrtimer_interrupt+0x128/0x2c0 [ 4720.287085] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4720.288264] ? apic_timer_interrupt+0xf/0x20 [ 4720.289312] [ 4720.303012] ? native_safe_halt+0x2e/0x50 [ 4720.304443] kvm_wait+0xcd/0xe0 [ 4720.305606] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4720.311939] do_raw_spin_lock+0x177/0x190 [ 4720.322155] _raw_spin_lock+0x21/0x30 [ 4720.326482] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4720.330915] ? do_raw_spin_unlock+0x75/0x190 [ 4720.333000] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4720.336446] ? do_wait_intr+0xf0/0xf0 [ 4720.337568] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4720.341584] mdc_reint+0x2d/0x110 [mdc] [ 4720.344316] mdc_unlink+0x658/0x870 [mdc] [ 4720.348678] lmv_unlink+0x34d/0xbd0 [lmv] [ 4720.352704] ll_unlink+0x24d/0x7f0 [lustre] [ 4720.354373] vfs_unlink+0x202/0x330 [ 4720.355527] do_unlinkat+0x2fa/0x460 [ 4720.358841] __x64_sys_unlink+0x37/0x50 [ 4720.361194] do_syscall_64+0xc1/0x440 [ 4720.364706] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4720.371099] RIP: 0033:0x7f83e5f5c10b [ 4720.373314] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4720.391117] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4720.396983] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4720.400699] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4720.412027] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4720.414558] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4720.418303] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4728.059012] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 4728.060344] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 4728.061773] Modules linked in: lustre(O) [ 4728.064376] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) [ 4728.071797] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4728.071872] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4728.071878] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4728.071881] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4728.071894] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4728.071898] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4728.071903] RAX: 000000000000549e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4728.071905] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4728.071907] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4728.071910] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4728.071912] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4728.071915] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4728.071918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4728.071920] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4728.071928] Call Trace: [ 4728.071932] [ 4728.071938] ? show_regs.cold.9+0x22/0x2f [ 4728.071948] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4728.071954] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4728.071959] ? watchdog+0x40/0x40 [ 4728.071963] ? hrtimer_interrupt+0x128/0x2c0 [ 4728.071967] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4728.071975] ? apic_timer_interrupt+0xf/0x20 [ 4728.071978] [ 4728.071981] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4728.071985] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4728.071991] do_raw_spin_lock+0x177/0x190 [ 4728.071995] _raw_spin_lock+0x21/0x30 [ 4728.072000] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4728.077776] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4728.077853] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4728.077860] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4728.077865] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4728.078190] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4728.078200] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4728.078204] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4728.078208] RAX: 0000000000004314 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4728.078210] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4728.078213] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4728.078215] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4728.078217] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4728.078221] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4728.078223] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4728.078226] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4728.078233] Call Trace: [ 4728.078238] [ 4728.079360] ? show_regs.cold.9+0x22/0x2f [ 4728.079376] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4728.079383] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4728.079387] ? watchdog+0x40/0x40 [ 4728.079390] ? hrtimer_interrupt+0x128/0x2c0 [ 4728.079394] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4728.079403] ? apic_timer_interrupt+0xf/0x20 [ 4728.079408] [ 4728.079411] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4728.079417] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4728.079422] do_raw_spin_lock+0x177/0x190 [ 4728.079435] _raw_spin_lock+0x21/0x30 [ 4728.079440] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4728.082869] process_one_work+0x2c8/0x700 [ 4728.082881] worker_thread+0x296/0x6e0 [ 4728.082886] ? rescuer_thread+0x570/0x570 [ 4728.082890] kthread+0x1d1/0x200 [ 4728.082894] ? set_kthread_struct+0x70/0x70 [ 4728.082898] ret_from_fork+0x1f/0x30 [ 4728.377662] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4728.380076] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4728.382426] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4728.384684] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4728.386895] ? woken_wake_function+0x30/0x30 [ 4728.388401] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4728.390455] kthread+0x1d1/0x200 [ 4728.391461] ? set_kthread_struct+0x70/0x70 [ 4728.393373] ret_from_fork+0x1f/0x30 [ 4732.060114] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 4732.065180] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4732.084044] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4732.088589] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4732.091970] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4732.094423] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4732.102171] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4732.105353] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4732.108239] RDX: 0000000000007018 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4732.110856] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4732.113762] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4732.116640] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4732.119652] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4732.122789] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4732.125434] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4732.127830] Call Trace: [ 4732.128837] [ 4732.129763] ? show_regs.cold.9+0x22/0x2f [ 4732.131492] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4732.133461] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4732.135401] ? watchdog+0x40/0x40 [ 4732.136562] ? hrtimer_interrupt+0x128/0x2c0 [ 4732.138339] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4732.139962] ? apic_timer_interrupt+0xf/0x20 [ 4732.141710] [ 4732.142638] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4732.144548] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4732.146637] do_raw_spin_lock+0x177/0x190 [ 4732.148174] _raw_spin_lock+0x21/0x30 [ 4732.149720] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4732.152226] obd_mod_free+0x46/0x140 [ptlrpc] [ 4732.154446] mdc_commit_open+0x91/0xb0 [mdc] [ 4732.156087] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4732.158483] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4732.160923] after_reply+0xe2c/0x1780 [ptlrpc] [ 4732.163059] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4732.165179] ? __next_timer_interrupt+0x160/0x160 [ 4732.166966] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4732.168827] ? do_wait_intr+0xf0/0xf0 [ 4732.170201] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4732.172532] kthread+0x1d1/0x200 [ 4732.173716] ? set_kthread_struct+0x70/0x70 [ 4732.175417] ret_from_fork+0x1f/0x30 [ 4736.993746] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2223s! [ 4737.012984] Showing busy workqueues and worker pools: [ 4737.027993] workqueue events: flags=0x0 [ 4737.040125] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4737.057049] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4737.070295] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4737.080374] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4737.091786] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4737.103010] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4737.122602] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4737.140060] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4737.149869] workqueue events_long: flags=0x0 [ 4737.164336] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4737.172719] pending: lu_objects_flush [obdclass] [ 4737.179864] workqueue events_unbound: flags=0x2 [ 4737.188729] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4737.201491] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4737.208794] workqueue events_power_efficient: flags=0x80 [ 4737.221387] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4737.229756] pending: neigh_periodic_work, do_cache_clean [ 4737.237398] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4737.242966] pending: neigh_periodic_work, check_lifetime [ 4737.248482] workqueue mm_percpu_wq: flags=0x8 [ 4737.258173] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4737.263520] pending: vmstat_update, lru_add_drain_per_cpu [ 4737.269536] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4737.278863] pending: vmstat_update [ 4737.284490] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4737.290565] pending: lru_add_drain_per_cpu BAR(48) [ 4737.297585] workqueue ipv6_addrconf: flags=0x40008 [ 4737.303744] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4737.310673] pending: addrconf_verify_work [ 4737.313924] workqueue knbd0-recv: flags=0x1a [ 4737.319423] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4737.325960] in-flight: 650:recv_work [ 4737.327950] workqueue ptlrpc_pinger: flags=0x2 [ 4737.335159] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4737.341688] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4737.344915] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2224s workers=4 idle: 36 466 29757 [ 4737.358752] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4727s workers=2 idle: 652 [ 4741.021574] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 4741.030374] rcu: 0-....: (1722413 ticks this GP) idle=30a/1/0x4000000000000000 softirq=128519/128519 fqs=126395 [ 4741.046442] (detected by 1, t=2760062 jiffies, g=262441, q=57956) [ 4741.055413] Sending NMI from CPU 1 to CPUs 0: [ 4741.063973] NMI backtrace for cpu 0 [ 4741.063980] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4741.063982] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4741.063984] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4741.063986] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4741.063988] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 [ 4741.063990] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4741.063991] RDX: 0000000000007d59 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4741.063993] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4741.063994] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4741.063995] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4741.063997] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4741.063998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4741.063999] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4741.064000] Call Trace: [ 4741.064001] [ 4741.064002] ? show_regs.cold.9+0x22/0x2f [ 4741.064003] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4741.064004] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4741.064005] ? nmi_handle+0xbd/0x210 [ 4741.064006] ? default_do_nmi+0x4b/0x1b0 [ 4741.064007] ? do_nmi+0x280/0x380 [ 4741.064008] ? end_repeat_nmi+0x16/0x1f [ 4741.064009] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4741.064010] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4741.064012] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4741.064013] [ 4741.064014] do_raw_spin_lock+0x177/0x190 [ 4741.064015] _raw_spin_lock+0x21/0x30 [ 4741.064016] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4741.064017] obd_mod_free+0x46/0x140 [ptlrpc] [ 4741.064018] mdc_commit_open+0x91/0xb0 [mdc] [ 4741.064020] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4741.064021] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4741.064022] after_reply+0xe2c/0x1780 [ptlrpc] [ 4741.064023] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4741.064024] ? __next_timer_interrupt+0x160/0x160 [ 4741.064025] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4741.064026] ? do_wait_intr+0xf0/0xf0 [ 4741.064027] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4741.064028] kthread+0x1d1/0x200 [ 4741.064029] ? set_kthread_struct+0x70/0x70 [ 4741.064030] ret_from_fork+0x1f/0x30 [ 4741.069152] rcu: rcu_sched kthread starved for 2229249 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 4741.412446] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4741.439921] rcu: RCU grace-period kthread stack dump: [ 4741.445842] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4741.463989] Call Trace: [ 4741.470868] __schedule+0x351/0xcb0 [ 4741.481139] schedule+0xc0/0x180 [ 4741.485873] schedule_timeout+0xb4/0x190 [ 4741.493123] ? do_raw_spin_unlock+0x75/0x190 [ 4741.501722] ? __next_timer_interrupt+0x160/0x160 [ 4741.515764] ? prepare_to_swait_event+0xf3/0x140 [ 4741.521142] rcu_gp_kthread+0x8dd/0x15b0 [ 4741.530597] ? force_qs_rnp+0x2c0/0x2c0 [ 4741.538428] kthread+0x1d1/0x200 [ 4741.542666] ? set_kthread_struct+0x70/0x70 [ 4741.548219] ret_from_fork+0x1f/0x30 [ 4741.552806] rcu: Stack dump where RCU GP kthread last ran: [ 4741.558395] Sending NMI from CPU 1 to CPUs 3: [ 4741.564566] NMI backtrace for cpu 3 [ 4741.564572] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4741.564575] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4741.564576] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 4741.564578] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 4741.564581] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 [ 4741.564583] RAX: 000000000000779d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4741.564585] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4741.564586] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4741.564588] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4741.564590] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4741.564591] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4741.564592] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4741.564594] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4741.564595] Call Trace: [ 4741.564596] [ 4741.564597] ? show_regs.cold.9+0x22/0x2f [ 4741.564598] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4741.564600] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4741.564601] ? nmi_handle+0xbd/0x210 [ 4741.564602] ? default_do_nmi+0x4b/0x1b0 [ 4741.564603] ? do_nmi+0x280/0x380 [ 4741.564604] ? end_repeat_nmi+0x16/0x1f [ 4741.564605] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 4741.564606] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 4741.564607] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 4741.564608] [ 4741.564609] do_raw_spin_lock+0x177/0x190 [ 4741.564610] _raw_spin_lock+0x21/0x30 [ 4741.564612] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4741.564613] ? do_raw_spin_unlock+0x75/0x190 [ 4741.564614] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4741.564615] ? do_wait_intr+0xf0/0xf0 [ 4741.564617] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4741.564618] mdc_reint+0x2d/0x110 [mdc] [ 4741.564619] mdc_unlink+0x658/0x870 [mdc] [ 4741.564620] lmv_unlink+0x34d/0xbd0 [lmv] [ 4741.564698] ll_unlink+0x24d/0x7f0 [lustre] [ 4741.564700] vfs_unlink+0x202/0x330 [ 4741.564701] do_unlinkat+0x2fa/0x460 [ 4741.564702] __x64_sys_unlink+0x37/0x50 [ 4741.564703] do_syscall_64+0xc1/0x440 [ 4741.564705] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4741.564706] RIP: 0033:0x7f83e5f5c10b [ 4741.564708] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4741.564710] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4741.564713] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4741.564715] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4741.564716] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4741.564718] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4741.564719] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4756.060109] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 4756.065604] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4756.127753] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4756.137946] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4756.142175] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4756.146424] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4756.148511] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4756.156185] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4756.159693] RAX: 0000000000005798 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4756.163219] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4756.166209] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4756.169346] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4756.181712] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4756.191899] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4756.200760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4756.212669] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4756.223961] Call Trace: [ 4756.226765] [ 4756.230724] ? show_regs.cold.9+0x22/0x2f [ 4756.239020] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4756.243470] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4756.250408] ? watchdog+0x40/0x40 [ 4756.254605] ? hrtimer_interrupt+0x128/0x2c0 [ 4756.264839] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4756.274257] ? apic_timer_interrupt+0xf/0x20 [ 4756.276751] [ 4756.278131] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4756.284682] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4756.291189] do_raw_spin_lock+0x177/0x190 [ 4756.295681] _raw_spin_lock+0x21/0x30 [ 4756.302723] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4756.312293] process_one_work+0x2c8/0x700 [ 4756.316429] worker_thread+0x296/0x6e0 [ 4756.319600] ? rescuer_thread+0x570/0x570 [ 4756.328545] kthread+0x1d1/0x200 [ 4756.333566] ? set_kthread_struct+0x70/0x70 [ 4756.336701] ret_from_fork+0x1f/0x30 [ 4767.716049] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2254s! [ 4767.720646] Showing busy workqueues and worker pools: [ 4767.723494] workqueue events: flags=0x0 [ 4767.725471] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4767.729086] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4767.733493] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4767.736846] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4767.740764] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4767.744388] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4767.750886] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4767.765685] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4767.775111] workqueue events_long: flags=0x0 [ 4767.781826] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4767.788766] pending: lu_objects_flush [obdclass] [ 4767.793816] workqueue events_unbound: flags=0x2 [ 4767.800498] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4767.805815] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4767.813009] workqueue events_power_efficient: flags=0x80 [ 4767.818445] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4767.824030] pending: neigh_periodic_work, do_cache_clean [ 4767.828105] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4767.832871] pending: neigh_periodic_work, check_lifetime [ 4767.837079] workqueue mm_percpu_wq: flags=0x8 [ 4767.839827] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4767.844200] pending: vmstat_update, lru_add_drain_per_cpu [ 4767.847381] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4767.851361] pending: vmstat_update [ 4767.853767] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4767.857848] pending: lru_add_drain_per_cpu BAR(48) [ 4767.861097] workqueue ipv6_addrconf: flags=0x40008 [ 4767.864722] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4767.870454] pending: addrconf_verify_work [ 4767.873171] workqueue knbd0-recv: flags=0x1a [ 4767.875624] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4767.882847] in-flight: 650:recv_work [ 4767.885033] workqueue ptlrpc_pinger: flags=0x2 [ 4767.889769] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4767.895691] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4767.899465] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2254s workers=4 idle: 36 466 29757 [ 4767.904767] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4757s workers=2 idle: 652 [ 4768.061826] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 4768.062176] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4768.074863] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4768.074949] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4768.074957] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4768.074959] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4768.074975] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4768.074979] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4768.074983] RAX: 0000000000006b9f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4768.074986] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4768.074988] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4768.074990] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4768.074992] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4768.074995] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4768.074998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4768.075000] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4768.075010] Call Trace: [ 4768.075015] [ 4768.075021] ? show_regs.cold.9+0x22/0x2f [ 4768.075032] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4768.075039] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4768.075043] ? watchdog+0x40/0x40 [ 4768.075047] ? hrtimer_interrupt+0x128/0x2c0 [ 4768.075051] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4768.075060] ? apic_timer_interrupt+0xf/0x20 [ 4768.075064] [ 4768.075066] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4768.075071] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4768.075178] do_raw_spin_lock+0x177/0x190 [ 4768.075184] _raw_spin_lock+0x21/0x30 [ 4768.075189] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4768.075964] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4768.080229] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) [ 4768.111060] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4768.116183] ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4768.116238] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4768.116246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4768.116249] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4768.116262] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4768.116267] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4768.116271] RAX: 000000000000620c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4768.116273] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4768.116276] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4768.116278] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4768.116280] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4768.116283] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4768.116286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4768.116288] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4768.116296] Call Trace: [ 4768.116301] [ 4768.116307] ? show_regs.cold.9+0x22/0x2f [ 4768.116316] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4768.116323] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4768.116327] ? watchdog+0x40/0x40 [ 4768.116331] ? hrtimer_interrupt+0x128/0x2c0 [ 4768.116335] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4768.116344] ? apic_timer_interrupt+0xf/0x20 [ 4768.116349] [ 4768.116351] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4768.116357] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4768.116361] do_raw_spin_lock+0x177/0x190 [ 4768.116367] _raw_spin_lock+0x21/0x30 [ 4768.116372] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4768.116640] ? do_raw_spin_unlock+0x75/0x190 [ 4768.116645] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4768.117378] ? do_wait_intr+0xf0/0xf0 [ 4768.117386] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4768.117579] mdc_reint+0x2d/0x110 [mdc] [ 4768.117616] mdc_unlink+0x658/0x870 [mdc] [ 4768.117643] lmv_unlink+0x34d/0xbd0 [lmv] [ 4768.117678] ll_unlink+0x24d/0x7f0 [lustre] [ 4768.117798] vfs_unlink+0x202/0x330 [ 4768.117806] do_unlinkat+0x2fa/0x460 [ 4768.117810] __x64_sys_unlink+0x37/0x50 [ 4768.117814] do_syscall_64+0xc1/0x440 [ 4768.117820] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4768.117828] RIP: 0033:0x7f83e5f5c10b [ 4768.117834] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4768.117838] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4768.117843] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4768.117845] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4768.117847] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4768.117849] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4768.117850] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4768.488239] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4768.490602] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4768.493433] ? woken_wake_function+0x30/0x30 [ 4768.495052] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4768.497537] kthread+0x1d1/0x200 [ 4768.498879] ? set_kthread_struct+0x70/0x70 [ 4768.500387] ret_from_fork+0x1f/0x30 [ 4784.065249] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 4784.082653] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4784.109300] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4784.114357] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4784.118439] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4784.121451] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4784.124112] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4784.142306] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4784.149849] RAX: 0000000000007e0c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4784.156795] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4784.162637] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4784.168041] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4784.171330] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4784.175402] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4784.178535] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4784.181254] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4784.185152] Call Trace: [ 4784.187886] [ 4784.189526] ? show_regs.cold.9+0x22/0x2f [ 4784.192682] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4784.196728] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4784.199463] ? watchdog+0x40/0x40 [ 4784.201877] ? hrtimer_interrupt+0x128/0x2c0 [ 4784.204466] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4784.208029] ? apic_timer_interrupt+0xf/0x20 [ 4784.210254] [ 4784.211286] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4784.214944] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4784.218391] do_raw_spin_lock+0x177/0x190 [ 4784.220292] _raw_spin_lock+0x21/0x30 [ 4784.222296] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4784.225401] process_one_work+0x2c8/0x700 [ 4784.227407] worker_thread+0x296/0x6e0 [ 4784.230002] ? rescuer_thread+0x570/0x570 [ 4784.232678] kthread+0x1d1/0x200 [ 4784.238060] ? set_kthread_struct+0x70/0x70 [ 4784.240087] ret_from_fork+0x1f/0x30 [ 4792.057356] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 4792.066829] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4792.124967] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4792.147884] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4792.156855] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4792.161148] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4792.178768] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4792.183914] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4792.187004] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4792.194535] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4792.198984] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4792.203648] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4792.206016] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4792.209898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4792.212046] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4792.215874] Call Trace: [ 4792.216891] [ 4792.217552] ? show_regs.cold.9+0x22/0x2f [ 4792.219824] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4792.222613] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4792.225878] ? watchdog+0x40/0x40 [ 4792.227555] ? hrtimer_interrupt+0x128/0x2c0 [ 4792.230088] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4792.231741] ? apic_timer_interrupt+0xf/0x20 [ 4792.240056] [ 4792.241731] ? native_safe_halt+0x2e/0x50 [ 4792.247892] kvm_wait+0xcd/0xe0 [ 4792.253492] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4792.257200] do_raw_spin_lock+0x177/0x190 [ 4792.260493] _raw_spin_lock+0x21/0x30 [ 4792.262432] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4792.268907] obd_mod_free+0x46/0x140 [ptlrpc] [ 4792.273794] mdc_commit_open+0x91/0xb0 [mdc] [ 4792.275273] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4792.282341] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4792.288921] after_reply+0xe2c/0x1780 [ptlrpc] [ 4792.292644] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4792.296612] ? __next_timer_interrupt+0x160/0x160 [ 4792.304811] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4792.310037] ? do_wait_intr+0xf0/0xf0 [ 4792.316925] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4792.320851] kthread+0x1d1/0x200 [ 4792.325481] ? set_kthread_struct+0x70/0x70 [ 4792.328219] ret_from_fork+0x1f/0x30 [ 4796.060803] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 4796.063404] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4796.070038] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4796.070113] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4796.070121] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4796.070124] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4796.070138] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4796.070142] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4796.070145] RAX: 0000000000000f07 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4796.070148] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4796.070150] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4796.070152] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4796.070155] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4796.070158] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4796.070160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4796.070162] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4796.070171] Call Trace: [ 4796.070176] [ 4796.070181] ? show_regs.cold.9+0x22/0x2f [ 4796.070190] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4796.070196] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4796.070200] ? watchdog+0x40/0x40 [ 4796.070204] ? hrtimer_interrupt+0x128/0x2c0 [ 4796.070207] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4796.070216] ? apic_timer_interrupt+0xf/0x20 [ 4796.070221] [ 4796.070223] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4796.070228] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4796.070234] do_raw_spin_lock+0x177/0x190 [ 4796.070239] _raw_spin_lock+0x21/0x30 [ 4796.070244] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4796.070656] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4796.070972] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4796.071203] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4796.078757] Modules linked in: lustre(O) [ 4796.124408] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4796.124751] ? woken_wake_function+0x30/0x30 [ 4796.124761] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4796.125163] kthread+0x1d1/0x200 [ 4796.125170] ? set_kthread_struct+0x70/0x70 [ 4796.125174] ret_from_fork+0x1f/0x30 [ 4796.312203] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4796.369149] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4796.382738] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4796.389025] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4796.393089] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4796.411279] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4796.420320] RAX: 0000000000000e91 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4796.426689] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4796.433015] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4796.437718] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4796.441523] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4796.447214] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4796.454381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4796.458391] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4796.463960] Call Trace: [ 4796.465381] [ 4796.470236] ? show_regs.cold.9+0x22/0x2f [ 4796.480286] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4796.488005] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4796.490189] ? watchdog+0x40/0x40 [ 4796.492515] ? hrtimer_interrupt+0x128/0x2c0 [ 4796.497173] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4796.499657] ? apic_timer_interrupt+0xf/0x20 [ 4796.501793] [ 4796.503874] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4796.510130] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4796.514314] do_raw_spin_lock+0x177/0x190 [ 4796.520545] _raw_spin_lock+0x21/0x30 [ 4796.524641] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4796.527979] ? do_raw_spin_unlock+0x75/0x190 [ 4796.530013] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4796.536985] ? do_wait_intr+0xf0/0xf0 [ 4796.539203] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4796.542610] mdc_reint+0x2d/0x110 [mdc] [ 4796.544831] mdc_unlink+0x658/0x870 [mdc] [ 4796.546618] lmv_unlink+0x34d/0xbd0 [lmv] [ 4796.554261] ll_unlink+0x24d/0x7f0 [lustre] [ 4796.558756] vfs_unlink+0x202/0x330 [ 4796.561543] do_unlinkat+0x2fa/0x460 [ 4796.568305] __x64_sys_unlink+0x37/0x50 [ 4796.573416] do_syscall_64+0xc1/0x440 [ 4796.580096] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4796.587134] RIP: 0033:0x7f83e5f5c10b [ 4796.588927] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4796.597346] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4796.600293] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4796.602739] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4796.605291] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4796.608472] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4796.611567] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4797.922453] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2284s! [ 4797.930044] Showing busy workqueues and worker pools: [ 4797.933071] workqueue events: flags=0x0 [ 4797.936552] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4797.940193] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4797.944696] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4797.947315] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4797.950152] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4797.952715] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4797.956022] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4797.958979] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4797.962676] workqueue events_long: flags=0x0 [ 4797.964347] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4797.967176] pending: lu_objects_flush [obdclass] [ 4797.969600] workqueue events_unbound: flags=0x2 [ 4797.971270] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4797.973584] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4797.976066] workqueue events_power_efficient: flags=0x80 [ 4797.978159] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4797.981082] pending: neigh_periodic_work, do_cache_clean [ 4797.983432] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4797.994080] pending: neigh_periodic_work, check_lifetime [ 4797.998888] workqueue mm_percpu_wq: flags=0x8 [ 4798.002823] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4798.010698] pending: vmstat_update, lru_add_drain_per_cpu [ 4798.012781] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4798.015914] pending: vmstat_update [ 4798.017540] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4798.020583] pending: lru_add_drain_per_cpu BAR(48) [ 4798.022865] workqueue ipv6_addrconf: flags=0x40008 [ 4798.026462] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4798.030979] pending: addrconf_verify_work [ 4798.033468] workqueue knbd0-recv: flags=0x1a [ 4798.034990] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4798.037470] in-flight: 650:recv_work [ 4798.039156] workqueue ptlrpc_pinger: flags=0x2 [ 4798.041105] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4798.043602] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4798.045988] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2284s workers=4 idle: 36 466 29757 [ 4798.050267] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4788s workers=2 idle: 652 [ 4812.061979] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 4812.066984] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4812.118864] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4812.131403] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4812.143987] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4812.151003] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4812.157908] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4812.174126] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4812.178586] RAX: 0000000000007680 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4812.192016] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4812.200304] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4812.211021] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4812.220959] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4812.232089] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4812.246806] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4812.254330] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4812.256731] Call Trace: [ 4812.259423] [ 4812.260365] ? show_regs.cold.9+0x22/0x2f [ 4812.262193] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4812.264425] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4812.266986] ? watchdog+0x40/0x40 [ 4812.268576] ? hrtimer_interrupt+0x128/0x2c0 [ 4812.274317] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4812.277239] ? apic_timer_interrupt+0xf/0x20 [ 4812.281822] [ 4812.282938] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4812.284962] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4812.289196] do_raw_spin_lock+0x177/0x190 [ 4812.291285] _raw_spin_lock+0x21/0x30 [ 4812.293074] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4812.296212] process_one_work+0x2c8/0x700 [ 4812.298063] worker_thread+0x296/0x6e0 [ 4812.300667] ? rescuer_thread+0x570/0x570 [ 4812.302425] kthread+0x1d1/0x200 [ 4812.305128] ? set_kthread_struct+0x70/0x70 [ 4812.307010] ret_from_fork+0x1f/0x30 [ 4824.057134] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 4824.060260] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4824.063382] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 4824.063904] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4824.063911] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4824.064210] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4824.064215] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4824.064217] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4824.064285] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4824.064290] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4824.064294] RAX: 0000000000001996 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4824.064297] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4824.064299] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4824.064301] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4824.064304] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4824.064308] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4824.064311] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4824.064385] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4824.064392] Call Trace: [ 4824.064409] [ 4824.064414] ? show_regs.cold.9+0x22/0x2f [ 4824.064423] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4824.064428] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4824.064432] ? watchdog+0x40/0x40 [ 4824.064435] ? hrtimer_interrupt+0x128/0x2c0 [ 4824.064439] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4824.064446] ? apic_timer_interrupt+0xf/0x20 [ 4824.064450] [ 4824.064452] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4824.064456] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4824.064460] do_raw_spin_lock+0x177/0x190 [ 4824.064464] _raw_spin_lock+0x21/0x30 [ 4824.064470] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4824.064718] ? do_raw_spin_unlock+0x75/0x190 [ 4824.064723] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4824.065198] ? do_wait_intr+0xf0/0xf0 [ 4824.065205] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4824.065460] mdc_reint+0x2d/0x110 [mdc] [ 4824.065495] mdc_unlink+0x658/0x870 [mdc] [ 4824.065520] lmv_unlink+0x34d/0xbd0 [lmv] [ 4824.065554] ll_unlink+0x24d/0x7f0 [lustre] [ 4824.065672] vfs_unlink+0x202/0x330 [ 4824.065679] do_unlinkat+0x2fa/0x460 [ 4824.065683] __x64_sys_unlink+0x37/0x50 [ 4824.065686] do_syscall_64+0xc1/0x440 [ 4824.065692] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4824.065697] RIP: 0033:0x7f83e5f5c10b [ 4824.065703] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4824.065706] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4824.065710] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4824.065712] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4824.065714] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4824.065716] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4824.065717] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4824.077605] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4824.077619] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4824.077622] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4824.077637] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4824.077641] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4824.077647] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4824.077649] RDX: 0000000000007a81 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4824.077651] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4824.077653] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4824.077656] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4824.077659] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4824.077662] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4824.077664] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4824.077672] Call Trace: [ 4824.077678] [ 4824.077688] ? show_regs.cold.9+0x22/0x2f [ 4824.077700] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4824.077707] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4824.077711] ? watchdog+0x40/0x40 [ 4824.077716] ? hrtimer_interrupt+0x128/0x2c0 [ 4824.077720] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4824.077727] ? apic_timer_interrupt+0xf/0x20 [ 4824.077733] [ 4824.077735] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4824.077740] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4824.077745] do_raw_spin_lock+0x177/0x190 [ 4824.077749] _raw_spin_lock+0x21/0x30 [ 4824.077756] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4824.078037] obd_mod_free+0x46/0x140 [ptlrpc] [ 4824.078263] mdc_commit_open+0x91/0xb0 [mdc] [ 4824.078467] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4824.078688] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4824.078901] after_reply+0xe2c/0x1780 [ptlrpc] [ 4824.079106] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4824.079319] ? __next_timer_interrupt+0x160/0x160 [ 4824.079329] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4824.079835] ? do_wait_intr+0xf0/0xf0 [ 4824.079844] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4824.080051] kthread+0x1d1/0x200 [ 4824.080057] ? set_kthread_struct+0x70/0x70 [ 4824.080061] ret_from_fork+0x1f/0x30 [ 4824.402137] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4824.419692] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4824.423836] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4824.426875] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4824.428948] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4824.440579] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4824.443518] RAX: 0000000000007d0b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4824.446465] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4824.449192] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4824.451676] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4824.454107] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4824.456586] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4824.460435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4824.462515] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4824.465083] Call Trace: [ 4824.466019] [ 4824.466733] ? show_regs.cold.9+0x22/0x2f [ 4824.468712] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4824.470447] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4824.472601] ? watchdog+0x40/0x40 [ 4824.473579] ? hrtimer_interrupt+0x128/0x2c0 [ 4824.475048] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4824.476545] ? apic_timer_interrupt+0xf/0x20 [ 4824.479797] [ 4824.480868] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4824.482746] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4824.484546] do_raw_spin_lock+0x177/0x190 [ 4824.485972] _raw_spin_lock+0x21/0x30 [ 4824.488049] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4824.491676] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4824.494106] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4824.499078] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4824.501855] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4824.504631] ? woken_wake_function+0x30/0x30 [ 4824.506086] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4824.508894] kthread+0x1d1/0x200 [ 4824.509988] ? set_kthread_struct+0x70/0x70 [ 4824.511325] ret_from_fork+0x1f/0x30 [ 4828.133726] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2314s! [ 4828.142775] Showing busy workqueues and worker pools: [ 4828.148505] workqueue events: flags=0x0 [ 4828.151669] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4828.155898] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4828.168203] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4828.175233] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4828.177803] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4828.184479] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4828.191746] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4828.199727] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4828.207039] workqueue events_long: flags=0x0 [ 4828.210527] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4828.214816] pending: lu_objects_flush [obdclass] [ 4828.222342] workqueue events_unbound: flags=0x2 [ 4828.225242] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4828.233498] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4828.239653] workqueue events_power_efficient: flags=0x80 [ 4828.244023] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4828.248949] pending: neigh_periodic_work, do_cache_clean [ 4828.259047] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4828.265867] pending: neigh_periodic_work, check_lifetime [ 4828.272855] workqueue mm_percpu_wq: flags=0x8 [ 4828.274462] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4828.285264] pending: vmstat_update, lru_add_drain_per_cpu [ 4828.289046] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4828.292031] pending: vmstat_update [ 4828.293592] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4828.298281] pending: lru_add_drain_per_cpu BAR(48) [ 4828.305457] workqueue ipv6_addrconf: flags=0x40008 [ 4828.309156] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4828.315961] pending: addrconf_verify_work [ 4828.317487] workqueue knbd0-recv: flags=0x1a [ 4828.318665] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4828.322330] in-flight: 650:recv_work [ 4828.325160] workqueue ptlrpc_pinger: flags=0x2 [ 4828.328494] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4828.333112] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4828.336257] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2315s workers=4 idle: 36 466 29757 [ 4828.340764] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4818s workers=2 idle: 652 [ 4840.060027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 4840.063564] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4840.085101] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4840.090091] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4840.093991] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4840.097026] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4840.099712] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4840.106839] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4840.109995] RAX: 000000000000061f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4840.112837] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4840.116301] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4840.119211] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4840.122710] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4840.125878] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4840.129371] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4840.131960] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4840.135743] Call Trace: [ 4840.136752] [ 4840.137731] ? show_regs.cold.9+0x22/0x2f [ 4840.139521] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4840.141573] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4840.143607] ? watchdog+0x40/0x40 [ 4840.145124] ? hrtimer_interrupt+0x128/0x2c0 [ 4840.146969] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4840.149148] ? apic_timer_interrupt+0xf/0x20 [ 4840.150825] [ 4840.151857] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4840.154438] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4840.156726] do_raw_spin_lock+0x177/0x190 [ 4840.158809] _raw_spin_lock+0x21/0x30 [ 4840.160656] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4840.163186] process_one_work+0x2c8/0x700 [ 4840.165053] worker_thread+0x296/0x6e0 [ 4840.166835] ? rescuer_thread+0x570/0x570 [ 4840.168618] kthread+0x1d1/0x200 [ 4840.169979] ? set_kthread_struct+0x70/0x70 [ 4840.171798] ret_from_fork+0x1f/0x30 [ 4852.061653] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 4852.062394] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4852.066944] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel [ 4852.070483] Modules linked in: [ 4852.070485] ata_piix serio_raw [ 4852.089609] lustre(O) [ 4852.091603] libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4852.091621] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4852.091628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4852.091631] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4852.091645] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4852.091649] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4852.091654] RAX: 0000000000005c60 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4852.091656] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4852.091658] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4852.091660] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4852.091662] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4852.091666] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4852.091669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4852.091671] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4852.091679] Call Trace: [ 4852.091684] [ 4852.091689] ? show_regs.cold.9+0x22/0x2f [ 4852.091699] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4852.091705] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4852.091709] ? watchdog+0x40/0x40 [ 4852.091713] ? hrtimer_interrupt+0x128/0x2c0 [ 4852.091717] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4852.091725] ? apic_timer_interrupt+0xf/0x20 [ 4852.091730] [ 4852.091732] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4852.091737] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4852.091744] do_raw_spin_lock+0x177/0x190 [ 4852.091748] _raw_spin_lock+0x21/0x30 [ 4852.091753] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4852.093493] osp(O) [ 4852.095517] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4852.095779] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4852.096396] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4852.101289] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4852.101367] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4852.101374] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4852.101377] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4852.101390] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4852.101395] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4852.101398] RAX: 0000000000003c4c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4852.101401] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4852.101403] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4852.101405] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4852.101407] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4852.101410] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4852.101412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4852.101414] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4852.101422] Call Trace: [ 4852.101427] [ 4852.101432] ? show_regs.cold.9+0x22/0x2f [ 4852.101441] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4852.101447] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4852.101451] ? watchdog+0x40/0x40 [ 4852.101455] ? hrtimer_interrupt+0x128/0x2c0 [ 4852.101463] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4852.101471] ? apic_timer_interrupt+0xf/0x20 [ 4852.101475] [ 4852.101477] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4852.101482] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4852.101485] do_raw_spin_lock+0x177/0x190 [ 4852.101490] _raw_spin_lock+0x21/0x30 [ 4852.101495] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4852.101772] ? do_raw_spin_unlock+0x75/0x190 [ 4852.101778] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4852.101993] ? do_wait_intr+0xf0/0xf0 [ 4852.102002] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4852.102235] mdc_reint+0x2d/0x110 [mdc] [ 4852.102272] mdc_unlink+0x658/0x870 [mdc] [ 4852.102306] lmv_unlink+0x34d/0xbd0 [lmv] [ 4852.102339] ll_unlink+0x24d/0x7f0 [lustre] [ 4852.102474] vfs_unlink+0x202/0x330 [ 4852.102482] do_unlinkat+0x2fa/0x460 [ 4852.102487] __x64_sys_unlink+0x37/0x50 [ 4852.102490] do_syscall_64+0xc1/0x440 [ 4852.102498] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4852.102503] RIP: 0033:0x7f83e5f5c10b [ 4852.102553] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4852.102558] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4852.102562] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4852.102565] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4852.102567] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4852.102569] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4852.102571] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4852.322971] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4852.325045] ? woken_wake_function+0x30/0x30 [ 4852.326993] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4852.329193] kthread+0x1d1/0x200 [ 4852.330479] ? set_kthread_struct+0x70/0x70 [ 4852.332462] ret_from_fork+0x1f/0x30 [ 4856.056022] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 4856.063699] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4856.096269] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4856.102556] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4856.107106] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4856.110008] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4856.119852] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4856.124176] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4856.128200] RDX: 0000000000002e6f RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4856.132225] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4856.136283] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4856.140487] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4856.144120] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4856.148224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4856.150494] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4856.154628] Call Trace: [ 4856.155906] [ 4856.157026] ? show_regs.cold.9+0x22/0x2f [ 4856.159148] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4856.161466] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4856.163765] ? watchdog+0x40/0x40 [ 4856.165392] ? hrtimer_interrupt+0x128/0x2c0 [ 4856.167363] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4856.170249] ? apic_timer_interrupt+0xf/0x20 [ 4856.172529] [ 4856.173525] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4856.176445] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4856.179491] do_raw_spin_lock+0x177/0x190 [ 4856.181634] _raw_spin_lock+0x21/0x30 [ 4856.184231] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4856.187940] obd_mod_free+0x46/0x140 [ptlrpc] [ 4856.191071] mdc_commit_open+0x91/0xb0 [mdc] [ 4856.193490] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4856.196891] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4856.200121] after_reply+0xe2c/0x1780 [ptlrpc] [ 4856.202899] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4856.205857] ? __next_timer_interrupt+0x160/0x160 [ 4856.208010] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4856.210613] ? do_wait_intr+0xf0/0xf0 [ 4856.212475] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4856.215553] kthread+0x1d1/0x200 [ 4856.217461] ? set_kthread_struct+0x70/0x70 [ 4856.219810] ret_from_fork+0x1f/0x30 [ 4858.849029] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2345s! [ 4858.852312] Showing busy workqueues and worker pools: [ 4858.855251] workqueue events: flags=0x0 [ 4858.856786] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4858.860729] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4858.864599] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4858.868666] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4858.870846] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4858.876582] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4858.879319] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4858.882352] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4858.890614] workqueue events_long: flags=0x0 [ 4858.893304] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4858.896249] pending: lu_objects_flush [obdclass] [ 4858.898650] workqueue events_unbound: flags=0x2 [ 4858.900598] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4858.903459] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4858.905623] workqueue events_power_efficient: flags=0x80 [ 4858.907544] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4858.910471] pending: neigh_periodic_work, do_cache_clean [ 4858.913203] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4858.915614] pending: neigh_periodic_work, check_lifetime [ 4858.917964] workqueue mm_percpu_wq: flags=0x8 [ 4858.919677] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4858.922490] pending: vmstat_update, lru_add_drain_per_cpu [ 4858.924754] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4858.929287] pending: vmstat_update [ 4858.930628] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4858.933407] pending: lru_add_drain_per_cpu BAR(48) [ 4858.937433] workqueue ipv6_addrconf: flags=0x40008 [ 4858.941032] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4858.943452] pending: addrconf_verify_work [ 4858.946414] workqueue knbd0-recv: flags=0x1a [ 4858.952486] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4858.958973] in-flight: 650:recv_work [ 4858.960994] workqueue ptlrpc_pinger: flags=0x2 [ 4858.962893] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4858.966329] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4858.971389] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2345s workers=4 idle: 36 466 29757 [ 4858.976162] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4849s workers=2 idle: 652 [ 4868.060018] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 4868.065680] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4868.089611] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4868.097783] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4868.101749] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4868.107343] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4868.110373] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4868.120030] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4868.123695] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4868.127377] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4868.130910] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4868.134276] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4868.136564] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4868.140280] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4868.144263] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4868.147508] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4868.151209] Call Trace: [ 4868.152549] [ 4868.153745] ? show_regs.cold.9+0x22/0x2f [ 4868.158066] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4868.161193] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4868.165259] ? watchdog+0x40/0x40 [ 4868.167133] ? hrtimer_interrupt+0x128/0x2c0 [ 4868.170461] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4868.173285] ? apic_timer_interrupt+0xf/0x20 [ 4868.175617] [ 4868.176525] ? native_safe_halt+0x2e/0x50 [ 4868.179545] kvm_wait+0xcd/0xe0 [ 4868.182286] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4868.186199] do_raw_spin_lock+0x177/0x190 [ 4868.190712] _raw_spin_lock+0x21/0x30 [ 4868.192687] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4868.195482] process_one_work+0x2c8/0x700 [ 4868.197758] worker_thread+0x296/0x6e0 [ 4868.199695] ? rescuer_thread+0x570/0x570 [ 4868.202219] kthread+0x1d1/0x200 [ 4868.204162] ? set_kthread_struct+0x70/0x70 [ 4868.207264] ret_from_fork+0x1f/0x30 [ 4880.062318] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 4880.065183] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4880.066593] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) [ 4880.074492] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4880.074581] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4880.074589] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4880.074592] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4880.074607] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4880.074623] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4880.074627] RAX: 0000000000006897 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4880.074630] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4880.074632] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4880.074633] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4880.074635] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4880.074638] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4880.074641] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4880.074643] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4880.074651] Call Trace: [ 4880.074657] [ 4880.074662] ? show_regs.cold.9+0x22/0x2f [ 4880.074671] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4880.074678] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4880.074682] ? watchdog+0x40/0x40 [ 4880.074687] ? hrtimer_interrupt+0x128/0x2c0 [ 4880.074691] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4880.074699] ? apic_timer_interrupt+0xf/0x20 [ 4880.074704] [ 4880.074706] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4880.074711] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4880.074716] do_raw_spin_lock+0x177/0x190 [ 4880.074721] _raw_spin_lock+0x21/0x30 [ 4880.074727] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4880.075310] ? do_raw_spin_unlock+0x75/0x190 [ 4880.075316] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4880.075543] ? do_wait_intr+0xf0/0xf0 [ 4880.075551] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4880.076094] mdc_reint+0x2d/0x110 [mdc] [ 4880.076132] mdc_unlink+0x658/0x870 [mdc] [ 4880.076165] lmv_unlink+0x34d/0xbd0 [lmv] [ 4880.095443] ptlrpc_gss(O) [ 4880.121369] ll_unlink+0x24d/0x7f0 [lustre] [ 4880.121493] vfs_unlink+0x202/0x330 [ 4880.121502] do_unlinkat+0x2fa/0x460 [ 4880.121507] __x64_sys_unlink+0x37/0x50 [ 4880.121511] do_syscall_64+0xc1/0x440 [ 4880.121518] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4880.121526] RIP: 0033:0x7f83e5f5c10b [ 4880.121533] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4880.121537] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4880.121542] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4880.121545] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4880.121547] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4880.121549] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4880.121551] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4880.376194] ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4880.399474] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4880.406577] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4880.409122] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4880.410787] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4880.417116] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4880.421409] RAX: 00000000000045e2 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4880.424027] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4880.426736] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4880.429384] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4880.432157] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4880.434428] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4880.438136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4880.440534] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4880.443833] Call Trace: [ 4880.444970] [ 4880.445672] ? show_regs.cold.9+0x22/0x2f [ 4880.447627] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4880.449588] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4880.451479] ? watchdog+0x40/0x40 [ 4880.453432] ? hrtimer_interrupt+0x128/0x2c0 [ 4880.457461] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4880.463349] ? apic_timer_interrupt+0xf/0x20 [ 4880.466712] [ 4880.469476] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4880.474101] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4880.477143] do_raw_spin_lock+0x177/0x190 [ 4880.480351] _raw_spin_lock+0x21/0x30 [ 4880.482201] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4880.485385] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4880.487580] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4880.490533] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4880.492411] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4880.496487] ? woken_wake_function+0x30/0x30 [ 4880.499331] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4880.503427] kthread+0x1d1/0x200 [ 4880.505218] ? set_kthread_struct+0x70/0x70 [ 4880.507098] ret_from_fork+0x1f/0x30 [ 4884.056647] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 4884.061925] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4884.089280] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4884.101629] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4884.108525] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4884.111266] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4884.133510] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4884.140947] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4884.150426] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4884.157168] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4884.165202] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4884.172936] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4884.182251] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4884.192601] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4884.199002] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4884.204298] Call Trace: [ 4884.205555] [ 4884.206557] ? show_regs.cold.9+0x22/0x2f [ 4884.210474] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4884.212327] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4884.217501] ? watchdog+0x40/0x40 [ 4884.221473] ? hrtimer_interrupt+0x128/0x2c0 [ 4884.224656] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4884.227757] ? apic_timer_interrupt+0xf/0x20 [ 4884.231006] [ 4884.235198] ? native_safe_halt+0x2e/0x50 [ 4884.240988] kvm_wait+0xcd/0xe0 [ 4884.242682] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4884.247443] do_raw_spin_lock+0x177/0x190 [ 4884.250298] _raw_spin_lock+0x21/0x30 [ 4884.253089] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4884.261006] obd_mod_free+0x46/0x140 [ptlrpc] [ 4884.270985] mdc_commit_open+0x91/0xb0 [mdc] [ 4884.272907] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4884.275394] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4884.278343] after_reply+0xe2c/0x1780 [ptlrpc] [ 4884.281069] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4884.284037] ? __next_timer_interrupt+0x160/0x160 [ 4884.286371] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4884.289717] ? do_wait_intr+0xf0/0xf0 [ 4884.291408] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4884.300521] kthread+0x1d1/0x200 [ 4884.304165] ? set_kthread_struct+0x70/0x70 [ 4884.306029] ret_from_fork+0x1f/0x30 [ 4889.064502] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2375s! [ 4889.077696] Showing busy workqueues and worker pools: [ 4889.083675] workqueue events: flags=0x0 [ 4889.088359] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4889.100519] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4889.108510] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4889.117560] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4889.122366] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4889.130686] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4889.145415] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4889.158660] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4889.172023] workqueue events_long: flags=0x0 [ 4889.175810] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4889.187268] pending: lu_objects_flush [obdclass] [ 4889.196483] workqueue events_unbound: flags=0x2 [ 4889.198092] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4889.206754] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4889.214869] workqueue events_power_efficient: flags=0x80 [ 4889.216794] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4889.226427] pending: neigh_periodic_work, do_cache_clean [ 4889.231890] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4889.240198] pending: neigh_periodic_work, check_lifetime [ 4889.249841] workqueue mm_percpu_wq: flags=0x8 [ 4889.254444] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4889.269607] pending: vmstat_update, lru_add_drain_per_cpu [ 4889.284645] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4889.292978] pending: vmstat_update [ 4889.298011] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4889.309104] pending: lru_add_drain_per_cpu BAR(48) [ 4889.320905] workqueue ipv6_addrconf: flags=0x40008 [ 4889.325327] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4889.336127] pending: addrconf_verify_work [ 4889.342262] workqueue knbd0-recv: flags=0x1a [ 4889.352361] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4889.366161] in-flight: 650:recv_work [ 4889.373097] workqueue ptlrpc_pinger: flags=0x2 [ 4889.381475] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4889.394248] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4889.411015] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2376s workers=4 idle: 36 466 29757 [ 4889.418294] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4879s workers=2 idle: 652 [ 4896.060657] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 4896.065583] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4896.082386] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4896.086801] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4896.089872] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4896.092166] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4896.094100] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4896.100337] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4896.102898] RAX: 00000000000012f9 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4896.105024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4896.107492] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4896.109925] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4896.112405] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4896.114926] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4896.117633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4896.119645] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4896.122145] Call Trace: [ 4896.123211] [ 4896.123899] ? show_regs.cold.9+0x22/0x2f [ 4896.125429] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4896.127202] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4896.128698] ? watchdog+0x40/0x40 [ 4896.129805] ? hrtimer_interrupt+0x128/0x2c0 [ 4896.131914] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4896.133521] ? apic_timer_interrupt+0xf/0x20 [ 4896.134940] [ 4896.135570] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4896.137296] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4896.139185] do_raw_spin_lock+0x177/0x190 [ 4896.140507] _raw_spin_lock+0x21/0x30 [ 4896.141678] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4896.144051] process_one_work+0x2c8/0x700 [ 4896.145511] worker_thread+0x296/0x6e0 [ 4896.146993] ? rescuer_thread+0x570/0x570 [ 4896.148388] kthread+0x1d1/0x200 [ 4896.149599] ? set_kthread_struct+0x70/0x70 [ 4896.151339] ret_from_fork+0x1f/0x30 [ 4908.059625] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 4908.063170] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4908.063560] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 4908.084121] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4908.084135] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4908.084139] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4908.084153] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4908.084158] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4908.084163] RAX: 0000000000000f07 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4908.084165] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4908.084167] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4908.084169] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4908.084172] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4908.084175] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4908.084178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4908.084180] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4908.084189] Call Trace: [ 4908.084194] [ 4908.084200] ? show_regs.cold.9+0x22/0x2f [ 4908.084208] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4908.084214] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4908.084219] ? watchdog+0x40/0x40 [ 4908.084223] ? hrtimer_interrupt+0x128/0x2c0 [ 4908.084227] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4908.084235] ? apic_timer_interrupt+0xf/0x20 [ 4908.084240] [ 4908.084244] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4908.084249] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4908.084257] do_raw_spin_lock+0x177/0x190 [ 4908.084261] _raw_spin_lock+0x21/0x30 [ 4908.084266] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4908.088683] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [ 4908.093380] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4908.093674] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4908.094145] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4908.094372] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4908.094596] ? woken_wake_function+0x30/0x30 [ 4908.094606] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4908.094829] kthread+0x1d1/0x200 [ 4908.094836] ? set_kthread_struct+0x70/0x70 [ 4908.094840] ret_from_fork+0x1f/0x30 [ 4908.319023] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4908.351360] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4908.367706] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4908.376290] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4908.384759] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4908.402047] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4908.409744] RAX: 000000000000744c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4908.414239] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4908.423053] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4908.429040] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4908.440774] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4908.443090] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4908.456503] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4908.461147] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4908.473327] Call Trace: [ 4908.478286] [ 4908.482940] ? show_regs.cold.9+0x22/0x2f [ 4908.487706] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4908.497519] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4908.503246] ? watchdog+0x40/0x40 [ 4908.507096] ? hrtimer_interrupt+0x128/0x2c0 [ 4908.519071] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4908.526908] ? apic_timer_interrupt+0xf/0x20 [ 4908.528513] [ 4908.529323] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4908.545913] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4908.550349] do_raw_spin_lock+0x177/0x190 [ 4908.555104] _raw_spin_lock+0x21/0x30 [ 4908.558235] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4908.561497] ? do_raw_spin_unlock+0x75/0x190 [ 4908.568224] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4908.577672] ? do_wait_intr+0xf0/0xf0 [ 4908.582214] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4908.591614] mdc_reint+0x2d/0x110 [mdc] [ 4908.596225] mdc_unlink+0x658/0x870 [mdc] [ 4908.601014] lmv_unlink+0x34d/0xbd0 [lmv] [ 4908.607571] ll_unlink+0x24d/0x7f0 [lustre] [ 4908.609031] vfs_unlink+0x202/0x330 [ 4908.611489] do_unlinkat+0x2fa/0x460 [ 4908.616839] __x64_sys_unlink+0x37/0x50 [ 4908.619687] do_syscall_64+0xc1/0x440 [ 4908.621028] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4908.637427] RIP: 0033:0x7f83e5f5c10b [ 4908.647357] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4908.666599] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4908.670497] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4908.675997] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4908.679131] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4908.687079] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4908.692316] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4916.057498] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 4916.063610] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4916.117516] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4916.124677] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4916.134006] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4916.136570] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4916.146147] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4916.149406] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4916.164151] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4916.178399] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4916.191678] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4916.201291] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4916.211012] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4916.217566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4916.220571] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4916.234796] Call Trace: [ 4916.235848] [ 4916.240093] ? show_regs.cold.9+0x22/0x2f [ 4916.241672] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4916.249963] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4916.254890] ? watchdog+0x40/0x40 [ 4916.262740] ? hrtimer_interrupt+0x128/0x2c0 [ 4916.266968] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4916.274207] ? apic_timer_interrupt+0xf/0x20 [ 4916.277648] [ 4916.279315] ? native_safe_halt+0x2e/0x50 [ 4916.285078] kvm_wait+0xcd/0xe0 [ 4916.289653] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4916.293772] do_raw_spin_lock+0x177/0x190 [ 4916.297326] _raw_spin_lock+0x21/0x30 [ 4916.301867] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4916.312420] obd_mod_free+0x46/0x140 [ptlrpc] [ 4916.319286] mdc_commit_open+0x91/0xb0 [mdc] [ 4916.320861] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4916.327819] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4916.336110] after_reply+0xe2c/0x1780 [ptlrpc] [ 4916.343442] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4916.350940] ? __next_timer_interrupt+0x160/0x160 [ 4916.358967] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4916.366930] ? do_wait_intr+0xf0/0xf0 [ 4916.368158] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4916.374398] kthread+0x1d1/0x200 [ 4916.379019] ? set_kthread_struct+0x70/0x70 [ 4916.385294] ret_from_fork+0x1f/0x30 [ 4919.776261] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2406s! [ 4919.783592] Showing busy workqueues and worker pools: [ 4919.787697] workqueue events: flags=0x0 [ 4919.789783] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4919.795346] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4919.802530] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4919.806789] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4919.817131] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4919.821587] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4919.829531] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4919.840586] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4919.853116] workqueue events_long: flags=0x0 [ 4919.859539] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4919.868476] pending: lu_objects_flush [obdclass] [ 4919.872802] workqueue events_unbound: flags=0x2 [ 4919.874691] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4919.883657] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4919.891375] workqueue events_power_efficient: flags=0x80 [ 4919.898170] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4919.907017] pending: neigh_periodic_work, do_cache_clean [ 4919.913264] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4919.920913] pending: neigh_periodic_work, check_lifetime [ 4919.927128] workqueue mm_percpu_wq: flags=0x8 [ 4919.932348] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4919.942999] pending: vmstat_update, lru_add_drain_per_cpu [ 4919.951481] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4919.958926] pending: vmstat_update [ 4919.962392] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4919.970419] pending: lru_add_drain_per_cpu BAR(48) [ 4919.979198] workqueue ipv6_addrconf: flags=0x40008 [ 4919.985575] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4919.991974] pending: addrconf_verify_work [ 4919.996749] workqueue knbd0-recv: flags=0x1a [ 4920.003017] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4920.015799] in-flight: 650:recv_work [ 4920.023774] workqueue ptlrpc_pinger: flags=0x2 [ 4920.029944] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4920.032695] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4920.038763] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2406s workers=4 idle: 36 466 29757 [ 4920.043024] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4910s workers=2 idle: 652 [ 4921.024115] rcu: INFO: rcu_sched self-detected stall on CPU [ 4921.033026] rcu: 0-....: (1834048 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 4921.041657] (t=2940082 jiffies g=262441 q=58008) [ 4921.043895] rcu: rcu_sched kthread starved for 2409271 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 4921.053136] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4921.065301] rcu: RCU grace-period kthread stack dump: [ 4921.072196] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4921.083060] Call Trace: [ 4921.085410] __schedule+0x351/0xcb0 [ 4921.088154] schedule+0xc0/0x180 [ 4921.089056] schedule_timeout+0xb4/0x190 [ 4921.091359] ? do_raw_spin_unlock+0x75/0x190 [ 4921.094381] ? __next_timer_interrupt+0x160/0x160 [ 4921.095915] ? prepare_to_swait_event+0xf3/0x140 [ 4921.106902] rcu_gp_kthread+0x8dd/0x15b0 [ 4921.110374] ? force_qs_rnp+0x2c0/0x2c0 [ 4921.114681] kthread+0x1d1/0x200 [ 4921.118814] ? set_kthread_struct+0x70/0x70 [ 4921.121951] ret_from_fork+0x1f/0x30 [ 4921.126034] rcu: Stack dump where RCU GP kthread last ran: [ 4921.128869] Sending NMI from CPU 0 to CPUs 3: [ 4921.131446] NMI backtrace for cpu 3 [ 4921.131452] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4921.131454] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4921.131455] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4921.131457] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4921.131459] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 [ 4921.131462] RAX: 000000000000000b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4921.131463] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4921.131465] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4921.131466] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4921.131467] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4921.131468] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4921.131470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4921.131471] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4921.131472] Call Trace: [ 4921.131473] [ 4921.131474] ? show_regs.cold.9+0x22/0x2f [ 4921.131475] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4921.131476] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4921.131477] ? nmi_handle+0xbd/0x210 [ 4921.131478] ? default_do_nmi+0x4b/0x1b0 [ 4921.131479] ? do_nmi+0x280/0x380 [ 4921.131481] ? end_repeat_nmi+0x16/0x1f [ 4921.131481] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4921.131482] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4921.131483] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4921.131484] [ 4921.131485] do_raw_spin_lock+0x177/0x190 [ 4921.131486] _raw_spin_lock+0x21/0x30 [ 4921.131487] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4921.131488] ? do_raw_spin_unlock+0x75/0x190 [ 4921.131488] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4921.131489] ? do_wait_intr+0xf0/0xf0 [ 4921.131490] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4921.131491] mdc_reint+0x2d/0x110 [mdc] [ 4921.131492] mdc_unlink+0x658/0x870 [mdc] [ 4921.131493] lmv_unlink+0x34d/0xbd0 [lmv] [ 4921.131494] ll_unlink+0x24d/0x7f0 [lustre] [ 4921.131495] vfs_unlink+0x202/0x330 [ 4921.131496] do_unlinkat+0x2fa/0x460 [ 4921.131497] __x64_sys_unlink+0x37/0x50 [ 4921.131498] do_syscall_64+0xc1/0x440 [ 4921.131499] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4921.131500] RIP: 0033:0x7f83e5f5c10b [ 4921.131501] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4921.131503] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4921.131505] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4921.131506] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4921.131508] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4921.131509] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4921.131510] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4921.141396] NMI backtrace for cpu 0 [ 4921.349557] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4921.355055] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4921.363399] Call Trace: [ 4921.364133] [ 4921.364738] ? dump_stack+0xbb/0x10e [ 4921.365665] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4921.370229] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4921.371483] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4921.374975] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4921.381282] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4921.383708] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4921.389050] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4921.392541] ? update_process_times+0x6d/0xc0 [ 4921.395098] ? tick_sched_handle.isra.18+0x27/0x90 [ 4921.397199] ? tick_sched_timer+0x53/0xb0 [ 4921.399616] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4921.404044] ? tick_sched_handle.isra.18+0x90/0x90 [ 4921.407636] ? hrtimer_interrupt+0x128/0x2c0 [ 4921.409585] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4921.411405] ? apic_timer_interrupt+0xf/0x20 [ 4921.420137] [ 4921.420688] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4921.422140] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4921.427311] ? do_raw_spin_lock+0x177/0x190 [ 4921.430194] ? _raw_spin_lock+0x21/0x30 [ 4921.431755] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4921.436966] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4921.440499] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4921.445682] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4921.448404] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4921.451384] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4921.455158] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4921.461816] ? __next_timer_interrupt+0x160/0x160 [ 4921.464309] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4921.466735] ? do_wait_intr+0xf0/0xf0 [ 4921.472718] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4921.476335] ? kthread+0x1d1/0x200 [ 4921.477747] ? set_kthread_struct+0x70/0x70 [ 4921.479746] ? ret_from_fork+0x1f/0x30 [ 4924.062726] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 4924.077068] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4924.150980] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4924.163160] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4924.167973] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4924.174199] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4924.177225] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4924.186597] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4924.189294] RAX: 0000000000007517 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4924.194556] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4924.198844] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4924.202475] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4924.205511] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4924.208828] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4924.214355] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4924.219075] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4924.223661] Call Trace: [ 4924.227498] [ 4924.228123] ? show_regs.cold.9+0x22/0x2f [ 4924.229196] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4924.232475] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4924.235516] ? watchdog+0x40/0x40 [ 4924.236379] ? hrtimer_interrupt+0x128/0x2c0 [ 4924.239040] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4924.242740] ? apic_timer_interrupt+0xf/0x20 [ 4924.249676] [ 4924.251319] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4924.253831] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4924.256261] do_raw_spin_lock+0x177/0x190 [ 4924.258320] _raw_spin_lock+0x21/0x30 [ 4924.261238] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4924.266238] process_one_work+0x2c8/0x700 [ 4924.269120] worker_thread+0x296/0x6e0 [ 4924.270259] ? rescuer_thread+0x570/0x570 [ 4924.272210] kthread+0x1d1/0x200 [ 4924.273991] ? set_kthread_struct+0x70/0x70 [ 4924.275676] ret_from_fork+0x1f/0x30 [ 4936.061574] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 4936.081569] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4936.112733] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4936.116357] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4936.122725] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4936.126280] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4936.133463] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4936.136628] RAX: 0000000000004995 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4936.139532] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4936.142469] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4936.145441] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4936.152703] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4936.158107] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4936.161027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4936.163778] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4936.166544] Call Trace: [ 4936.167853] [ 4936.168731] ? show_regs.cold.9+0x22/0x2f [ 4936.174885] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4936.181333] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4936.185589] ? watchdog+0x40/0x40 [ 4936.188221] ? hrtimer_interrupt+0x128/0x2c0 [ 4936.191561] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4936.197622] ? apic_timer_interrupt+0xf/0x20 [ 4936.200907] [ 4936.202141] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4936.206103] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4936.211049] do_raw_spin_lock+0x177/0x190 [ 4936.212865] _raw_spin_lock+0x21/0x30 [ 4936.214525] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4936.222569] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4936.226047] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4936.228857] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4936.231467] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4936.233848] ? woken_wake_function+0x30/0x30 [ 4936.235770] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4936.240171] kthread+0x1d1/0x200 [ 4936.241398] ? set_kthread_struct+0x70/0x70 [ 4936.245898] ret_from_fork+0x1f/0x30 [ 4948.056035] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 4948.059640] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4948.062120] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 4948.094527] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4948.098748] Modules linked in: [ 4948.109700] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4948.109711] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4948.109725] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4948.109729] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4948.109735] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4948.109737] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4948.109739] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4948.109741] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4948.109744] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4948.109747] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4948.109750] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4948.109752] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4948.109761] Call Trace: [ 4948.109766] [ 4948.109772] ? show_regs.cold.9+0x22/0x2f [ 4948.109782] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4948.109789] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4948.109793] ? watchdog+0x40/0x40 [ 4948.109798] ? hrtimer_interrupt+0x128/0x2c0 [ 4948.109802] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4948.109808] ? apic_timer_interrupt+0xf/0x20 [ 4948.109813] [ 4948.109816] ? native_safe_halt+0x2e/0x50 [ 4948.109820] kvm_wait+0xcd/0xe0 [ 4948.109827] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4948.110109] do_raw_spin_lock+0x177/0x190 [ 4948.110117] _raw_spin_lock+0x21/0x30 [ 4948.110123] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4948.110552] obd_mod_free+0x46/0x140 [ptlrpc] [ 4948.110749] mdc_commit_open+0x91/0xb0 [mdc] [ 4948.110784] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4948.111331] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4948.111822] after_reply+0xe2c/0x1780 [ptlrpc] [ 4948.112286] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4948.112545] ? __next_timer_interrupt+0x160/0x160 [ 4948.112554] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4948.112747] ? do_wait_intr+0xf0/0xf0 [ 4948.112803] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4948.113288] kthread+0x1d1/0x200 [ 4948.113294] ? set_kthread_struct+0x70/0x70 [ 4948.113297] ret_from_fork+0x1f/0x30 [ 4948.333286] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4948.376409] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4948.384424] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4948.387373] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4948.392697] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4948.408088] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4948.412375] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4948.422297] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4948.429280] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4948.433023] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4948.443536] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4948.448471] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4948.455678] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4948.465497] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4948.469780] Call Trace: [ 4948.472698] [ 4948.476435] ? show_regs.cold.9+0x22/0x2f [ 4948.478695] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4948.483538] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4948.485569] ? watchdog+0x40/0x40 [ 4948.488595] ? hrtimer_interrupt+0x128/0x2c0 [ 4948.498127] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4948.502310] ? apic_timer_interrupt+0xf/0x20 [ 4948.506747] [ 4948.507807] ? native_safe_halt+0x2e/0x50 [ 4948.510830] kvm_wait+0xcd/0xe0 [ 4948.512676] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4948.515621] do_raw_spin_lock+0x177/0x190 [ 4948.521101] _raw_spin_lock+0x21/0x30 [ 4948.523027] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4948.527090] ? do_raw_spin_unlock+0x75/0x190 [ 4948.530279] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4948.534774] ? do_wait_intr+0xf0/0xf0 [ 4948.538041] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4948.539837] mdc_reint+0x2d/0x110 [mdc] [ 4948.541808] mdc_unlink+0x658/0x870 [mdc] [ 4948.544119] lmv_unlink+0x34d/0xbd0 [lmv] [ 4948.545605] ll_unlink+0x24d/0x7f0 [lustre] [ 4948.547611] vfs_unlink+0x202/0x330 [ 4948.548705] do_unlinkat+0x2fa/0x460 [ 4948.551896] __x64_sys_unlink+0x37/0x50 [ 4948.552975] do_syscall_64+0xc1/0x440 [ 4948.553942] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4948.559383] RIP: 0033:0x7f83e5f5c10b [ 4948.562126] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4948.571618] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4948.575473] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4948.583632] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4948.590601] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4948.597319] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4948.610589] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4950.496357] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2437s! [ 4950.510182] Showing busy workqueues and worker pools: [ 4950.516759] workqueue events: flags=0x0 [ 4950.520666] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4950.528748] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4950.541451] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4950.547700] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4950.555283] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4950.561035] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4950.569036] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4950.579828] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4950.585171] workqueue events_long: flags=0x0 [ 4950.591028] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4950.597336] pending: lu_objects_flush [obdclass] [ 4950.598884] workqueue events_unbound: flags=0x2 [ 4950.600504] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4950.608389] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4950.616820] workqueue events_power_efficient: flags=0x80 [ 4950.620804] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4950.626945] pending: neigh_periodic_work, do_cache_clean [ 4950.630052] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4950.637725] pending: neigh_periodic_work, check_lifetime [ 4950.643831] workqueue mm_percpu_wq: flags=0x8 [ 4950.648881] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4950.654881] pending: vmstat_update, lru_add_drain_per_cpu [ 4950.657532] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4950.668522] pending: vmstat_update [ 4950.670842] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4950.673043] pending: lru_add_drain_per_cpu BAR(48) [ 4950.675549] workqueue ipv6_addrconf: flags=0x40008 [ 4950.677829] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4950.685737] pending: addrconf_verify_work [ 4950.694158] workqueue knbd0-recv: flags=0x1a [ 4950.695565] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4950.703530] in-flight: 650:recv_work [ 4950.709971] workqueue ptlrpc_pinger: flags=0x2 [ 4950.711660] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4950.717584] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4950.724272] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2437s workers=4 idle: 36 466 29757 [ 4950.739503] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4940s workers=2 idle: 652 [ 4952.062658] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 4952.079070] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4952.138737] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4952.153303] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4952.163108] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4952.170788] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4952.172849] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4952.201259] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4952.209270] RAX: 0000000000005f0b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4952.214769] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4952.229510] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4952.237398] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4952.239607] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4952.251027] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4952.268854] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4952.273296] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4952.283323] Call Trace: [ 4952.284959] [ 4952.286452] ? show_regs.cold.9+0x22/0x2f [ 4952.292401] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4952.296731] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4952.299887] ? watchdog+0x40/0x40 [ 4952.301183] ? hrtimer_interrupt+0x128/0x2c0 [ 4952.307551] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4952.313760] ? apic_timer_interrupt+0xf/0x20 [ 4952.325082] [ 4952.325783] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4952.337007] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4952.345376] do_raw_spin_lock+0x177/0x190 [ 4952.348363] _raw_spin_lock+0x21/0x30 [ 4952.352329] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4952.360318] process_one_work+0x2c8/0x700 [ 4952.367235] worker_thread+0x296/0x6e0 [ 4952.373542] ? rescuer_thread+0x570/0x570 [ 4952.382246] kthread+0x1d1/0x200 [ 4952.386429] ? set_kthread_struct+0x70/0x70 [ 4952.388044] ret_from_fork+0x1f/0x30 [ 4964.060999] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 4964.071448] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4964.125584] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4964.144119] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4964.149911] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4964.151452] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4964.177299] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4964.190265] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4964.197346] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4964.211232] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4964.224128] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4964.235772] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4964.245485] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4964.250529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4964.263212] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4964.275870] Call Trace: [ 4964.279848] [ 4964.283898] ? show_regs.cold.9+0x22/0x2f [ 4964.285195] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4964.286603] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4964.298270] ? watchdog+0x40/0x40 [ 4964.300509] ? hrtimer_interrupt+0x128/0x2c0 [ 4964.306080] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4964.313142] ? apic_timer_interrupt+0xf/0x20 [ 4964.319283] [ 4964.323062] ? native_safe_halt+0x2e/0x50 [ 4964.326364] kvm_wait+0xcd/0xe0 [ 4964.333091] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4964.339511] do_raw_spin_lock+0x177/0x190 [ 4964.344154] _raw_spin_lock+0x21/0x30 [ 4964.345290] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4964.357158] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4964.361526] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4964.364032] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4964.365773] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4964.370197] ? woken_wake_function+0x30/0x30 [ 4964.374221] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4964.383892] kthread+0x1d1/0x200 [ 4964.390916] ? set_kthread_struct+0x70/0x70 [ 4964.395971] ret_from_fork+0x1f/0x30 [ 4976.056049] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 4976.058582] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4976.062017] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 4976.081940] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4976.081951] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4976.081953] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4976.081963] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4976.081966] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4976.081970] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 4976.081972] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 4976.081973] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 4976.081976] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 4976.081978] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4976.081981] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 4976.081984] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4976.081986] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 4976.081994] Call Trace: [ 4976.081999] [ 4976.082005] ? show_regs.cold.9+0x22/0x2f [ 4976.082015] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4976.082021] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4976.082026] ? watchdog+0x40/0x40 [ 4976.082030] ? hrtimer_interrupt+0x128/0x2c0 [ 4976.082033] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4976.082039] ? apic_timer_interrupt+0xf/0x20 [ 4976.082042] [ 4976.082044] ? native_safe_halt+0x2e/0x50 [ 4976.082047] kvm_wait+0xcd/0xe0 [ 4976.082052] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4976.082057] do_raw_spin_lock+0x177/0x190 [ 4976.082060] _raw_spin_lock+0x21/0x30 [ 4976.082064] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4976.082259] obd_mod_free+0x46/0x140 [ptlrpc] [ 4976.082383] mdc_commit_open+0x91/0xb0 [mdc] [ 4976.082411] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4976.082554] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4976.082692] after_reply+0xe2c/0x1780 [ptlrpc] [ 4976.084720] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4976.084794] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4976.084801] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4976.084804] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4976.084815] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4976.084819] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4976.084823] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4976.084826] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 4976.084828] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4976.084829] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 4976.084832] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 4976.084835] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 4976.084838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4976.084840] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 4976.084847] Call Trace: [ 4976.084850] [ 4976.084854] ? show_regs.cold.9+0x22/0x2f [ 4976.084861] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4976.084867] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4976.084871] ? watchdog+0x40/0x40 [ 4976.084874] ? hrtimer_interrupt+0x128/0x2c0 [ 4976.084878] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4976.084894] ? apic_timer_interrupt+0xf/0x20 [ 4976.084898] [ 4976.084901] ? native_safe_halt+0x2e/0x50 [ 4976.084904] kvm_wait+0xcd/0xe0 [ 4976.084911] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4976.084917] do_raw_spin_lock+0x177/0x190 [ 4976.084921] _raw_spin_lock+0x21/0x30 [ 4976.084926] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4976.085159] ? do_raw_spin_unlock+0x75/0x190 [ 4976.085163] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4976.085287] ? do_wait_intr+0xf0/0xf0 [ 4976.085293] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4976.085418] mdc_reint+0x2d/0x110 [mdc] [ 4976.085444] mdc_unlink+0x658/0x870 [mdc] [ 4976.085467] lmv_unlink+0x34d/0xbd0 [lmv] [ 4976.085498] ll_unlink+0x24d/0x7f0 [lustre] [ 4976.085581] vfs_unlink+0x202/0x330 [ 4976.085586] do_unlinkat+0x2fa/0x460 [ 4976.085590] __x64_sys_unlink+0x37/0x50 [ 4976.085591] do_syscall_64+0xc1/0x440 [ 4976.085596] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4976.085600] RIP: 0033:0x7f83e5f5c10b [ 4976.085604] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 4976.085606] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 4976.085609] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 4976.085611] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 4976.085613] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 4976.085614] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 4976.085615] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 4976.311453] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4976.313551] ? __next_timer_interrupt+0x160/0x160 [ 4976.315422] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4976.317255] ? do_wait_intr+0xf0/0xf0 [ 4976.318628] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4976.320672] kthread+0x1d1/0x200 [ 4976.321776] ? set_kthread_struct+0x70/0x70 [ 4976.323363] ret_from_fork+0x1f/0x30 [ 4980.060328] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 4980.064507] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4980.131742] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4980.144432] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4980.156243] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4980.158385] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4980.162130] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4980.182631] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4980.193637] RAX: 000000000000678b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4980.200530] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 4980.219328] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 4980.222743] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 4980.231175] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 4980.235399] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 4980.241682] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4980.247859] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 4980.250243] Call Trace: [ 4980.253223] [ 4980.254286] ? show_regs.cold.9+0x22/0x2f [ 4980.262808] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4980.269101] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4980.275856] ? watchdog+0x40/0x40 [ 4980.279168] ? hrtimer_interrupt+0x128/0x2c0 [ 4980.287379] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4980.294040] ? apic_timer_interrupt+0xf/0x20 [ 4980.299847] [ 4980.303230] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4980.311625] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4980.316801] do_raw_spin_lock+0x177/0x190 [ 4980.320758] _raw_spin_lock+0x21/0x30 [ 4980.324671] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4980.327503] process_one_work+0x2c8/0x700 [ 4980.336318] worker_thread+0x296/0x6e0 [ 4980.343262] ? rescuer_thread+0x570/0x570 [ 4980.349084] kthread+0x1d1/0x200 [ 4980.350819] ? set_kthread_struct+0x70/0x70 [ 4980.353107] ret_from_fork+0x1f/0x30 [ 4981.217856] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2468s! [ 4981.226981] Showing busy workqueues and worker pools: [ 4981.235517] workqueue events: flags=0x0 [ 4981.238097] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4981.239781] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 4981.242410] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4981.244787] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 4981.247428] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4981.249562] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 4981.252035] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4981.254554] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 4981.258147] workqueue events_long: flags=0x0 [ 4981.259766] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4981.262242] pending: lu_objects_flush [obdclass] [ 4981.264179] workqueue events_unbound: flags=0x2 [ 4981.265789] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4981.268220] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4981.270263] workqueue events_power_efficient: flags=0x80 [ 4981.272351] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4981.275043] pending: neigh_periodic_work, do_cache_clean [ 4981.277046] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4981.279553] pending: neigh_periodic_work, check_lifetime [ 4981.281616] workqueue mm_percpu_wq: flags=0x8 [ 4981.283225] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4981.284965] pending: vmstat_update, lru_add_drain_per_cpu [ 4981.286657] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4981.289373] pending: vmstat_update [ 4981.290428] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 4981.292823] pending: lru_add_drain_per_cpu BAR(48) [ 4981.294799] workqueue ipv6_addrconf: flags=0x40008 [ 4981.296361] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4981.298698] pending: addrconf_verify_work [ 4981.300341] workqueue knbd0-recv: flags=0x1a [ 4981.302051] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4981.304612] in-flight: 650:recv_work [ 4981.306198] workqueue ptlrpc_pinger: flags=0x2 [ 4981.307982] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4981.310226] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 4981.312625] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2468s workers=4 idle: 36 466 29757 [ 4981.315413] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4971s workers=2 idle: 652 [ 4992.060799] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 4992.073862] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4992.126596] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4992.139125] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4992.147145] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4992.151098] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4992.165844] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4992.169231] RAX: 0000000000006786 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 4992.178919] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 4992.189195] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4992.198079] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 4992.201476] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 4992.207077] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 4992.210567] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4992.214179] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 4992.222140] Call Trace: [ 4992.223904] [ 4992.224825] ? show_regs.cold.9+0x22/0x2f [ 4992.226636] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4992.229486] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4992.231425] ? watchdog+0x40/0x40 [ 4992.232763] ? hrtimer_interrupt+0x128/0x2c0 [ 4992.236076] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4992.241334] ? apic_timer_interrupt+0xf/0x20 [ 4992.245876] [ 4992.246895] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4992.250630] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4992.257731] do_raw_spin_lock+0x177/0x190 [ 4992.260086] _raw_spin_lock+0x21/0x30 [ 4992.262464] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4992.265896] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4992.269423] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4992.272218] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4992.278040] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4992.282659] ? woken_wake_function+0x30/0x30 [ 4992.285011] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4992.287957] kthread+0x1d1/0x200 [ 4992.289713] ? set_kthread_struct+0x70/0x70 [ 4992.291857] ret_from_fork+0x1f/0x30 [ 5004.063778] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5004.071399] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5004.123703] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5004.128395] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5004.148155] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5004.154057] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5004.170993] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5004.181810] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5004.187233] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5004.194682] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5004.200944] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5004.204631] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5004.208064] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5004.216639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5004.224483] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5004.237746] Call Trace: [ 5004.241471] [ 5004.245752] ? show_regs.cold.9+0x22/0x2f [ 5004.247567] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5004.249242] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5004.257286] ? watchdog+0x40/0x40 [ 5004.258489] ? hrtimer_interrupt+0x128/0x2c0 [ 5004.263737] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5004.266038] ? apic_timer_interrupt+0xf/0x20 [ 5004.269961] [ 5004.271078] ? native_safe_halt+0x2e/0x50 [ 5004.273389] kvm_wait+0xcd/0xe0 [ 5004.274781] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5004.277091] do_raw_spin_lock+0x177/0x190 [ 5004.278945] _raw_spin_lock+0x21/0x30 [ 5004.283050] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5004.294877] ? do_raw_spin_unlock+0x75/0x190 [ 5004.304101] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5004.312066] ? do_wait_intr+0xf0/0xf0 [ 5004.318257] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5004.324080] mdc_reint+0x2d/0x110 [mdc] [ 5004.329254] mdc_unlink+0x658/0x870 [mdc] [ 5004.332086] lmv_unlink+0x34d/0xbd0 [lmv] [ 5004.339569] ll_unlink+0x24d/0x7f0 [lustre] [ 5004.343749] vfs_unlink+0x202/0x330 [ 5004.348454] do_unlinkat+0x2fa/0x460 [ 5004.352926] __x64_sys_unlink+0x37/0x50 [ 5004.362337] do_syscall_64+0xc1/0x440 [ 5004.372646] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5004.386002] RIP: 0033:0x7f83e5f5c10b [ 5004.389699] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5004.425067] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5004.428153] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5004.433982] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5004.438162] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5004.445972] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5004.453218] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5008.060059] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 5008.064124] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 5008.065230] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5008.065307] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5008.065314] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5008.065317] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5008.065334] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5008.065338] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5008.065342] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5008.065345] RDX: 000000000000557e RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5008.065347] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5008.065349] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5008.065352] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5008.065355] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5008.065358] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5008.065360] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5008.065377] Call Trace: [ 5008.065382] [ 5008.065388] ? show_regs.cold.9+0x22/0x2f [ 5008.065398] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5008.065405] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5008.065409] ? watchdog+0x40/0x40 [ 5008.065414] ? hrtimer_interrupt+0x128/0x2c0 [ 5008.065418] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5008.065426] ? apic_timer_interrupt+0xf/0x20 [ 5008.065431] [ 5008.065433] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5008.065439] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5008.065443] do_raw_spin_lock+0x177/0x190 [ 5008.065448] _raw_spin_lock+0x21/0x30 [ 5008.065454] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5008.065722] obd_mod_free+0x46/0x140 [ptlrpc] [ 5008.065913] mdc_commit_open+0x91/0xb0 [mdc] [ 5008.065948] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5008.066151] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5008.066348] after_reply+0xe2c/0x1780 [ptlrpc] [ 5008.067864] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5008.068076] ? __next_timer_interrupt+0x160/0x160 [ 5008.068089] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5008.068691] ? do_wait_intr+0xf0/0xf0 [ 5008.068701] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5008.068909] kthread+0x1d1/0x200 [ 5008.068915] ? set_kthread_struct+0x70/0x70 [ 5008.068919] ret_from_fork+0x1f/0x30 [ 5008.374145] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5008.433755] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5008.444024] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5008.452889] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5008.456579] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5008.465044] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5008.479565] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5008.492123] RAX: 0000000000007455 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5008.499263] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5008.515766] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5008.523155] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5008.531433] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5008.539726] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5008.549458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5008.555106] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5008.570073] Call Trace: [ 5008.573895] [ 5008.578631] ? show_regs.cold.9+0x22/0x2f [ 5008.584432] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5008.590222] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5008.597811] ? watchdog+0x40/0x40 [ 5008.605152] ? hrtimer_interrupt+0x128/0x2c0 [ 5008.606976] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5008.618994] ? apic_timer_interrupt+0xf/0x20 [ 5008.625627] [ 5008.629233] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5008.634300] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5008.638089] do_raw_spin_lock+0x177/0x190 [ 5008.640854] _raw_spin_lock+0x21/0x30 [ 5008.643092] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5008.646778] process_one_work+0x2c8/0x700 [ 5008.649976] worker_thread+0x296/0x6e0 [ 5008.652450] ? rescuer_thread+0x570/0x570 [ 5008.654803] kthread+0x1d1/0x200 [ 5008.661959] ? set_kthread_struct+0x70/0x70 [ 5008.668124] ret_from_fork+0x1f/0x30 [ 5011.424554] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2498s! [ 5011.430926] Showing busy workqueues and worker pools: [ 5011.436472] workqueue events: flags=0x0 [ 5011.440527] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5011.446179] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5011.456452] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5011.467861] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5011.475694] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5011.481470] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5011.492882] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5011.498477] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5011.505637] workqueue events_long: flags=0x0 [ 5011.508993] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5011.519437] pending: lu_objects_flush [obdclass] [ 5011.527135] workqueue events_unbound: flags=0x2 [ 5011.531196] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5011.546940] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5011.560169] workqueue events_power_efficient: flags=0x80 [ 5011.574373] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5011.583997] pending: neigh_periodic_work, do_cache_clean [ 5011.589114] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5011.597315] pending: neigh_periodic_work, check_lifetime [ 5011.599476] workqueue mm_percpu_wq: flags=0x8 [ 5011.605730] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5011.623618] pending: vmstat_update, lru_add_drain_per_cpu [ 5011.625645] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5011.638358] pending: vmstat_update [ 5011.646881] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5011.658449] pending: lru_add_drain_per_cpu BAR(48) [ 5011.668454] workqueue ipv6_addrconf: flags=0x40008 [ 5011.672643] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5011.676704] pending: addrconf_verify_work [ 5011.681674] workqueue knbd0-recv: flags=0x1a [ 5011.683475] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5011.689780] in-flight: 650:recv_work [ 5011.697689] workqueue ptlrpc_pinger: flags=0x2 [ 5011.699382] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5011.704185] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5011.707607] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2498s workers=4 idle: 36 466 29757 [ 5011.715961] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5001s workers=2 idle: 652 [ 5020.059026] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 5020.061436] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5020.100954] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5020.112878] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5020.122965] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5020.128835] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5020.153444] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5020.161844] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5020.167283] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5020.172647] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5020.178082] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5020.179640] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5020.182733] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5020.185312] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5020.187136] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5020.191564] Call Trace: [ 5020.192929] [ 5020.193941] ? show_regs.cold.9+0x22/0x2f [ 5020.195955] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5020.198397] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5020.200524] ? watchdog+0x40/0x40 [ 5020.201821] ? hrtimer_interrupt+0x128/0x2c0 [ 5020.203449] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5020.205534] ? apic_timer_interrupt+0xf/0x20 [ 5020.207291] [ 5020.208571] ? native_safe_halt+0x2e/0x50 [ 5020.209954] kvm_wait+0xcd/0xe0 [ 5020.210967] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5020.213569] do_raw_spin_lock+0x177/0x190 [ 5020.217889] _raw_spin_lock+0x21/0x30 [ 5020.220167] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5020.223786] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5020.230977] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5020.240153] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5020.249411] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5020.253277] ? woken_wake_function+0x30/0x30 [ 5020.255559] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5020.259154] kthread+0x1d1/0x200 [ 5020.260464] ? set_kthread_struct+0x70/0x70 [ 5020.263366] ret_from_fork+0x1f/0x30 [ 5032.074076] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5032.083743] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5032.164985] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5032.182984] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5032.189265] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5032.196422] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5032.222667] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5032.232301] RAX: 000000000000640f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5032.241609] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5032.250096] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5032.261599] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5032.264055] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5032.275036] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5032.286868] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5032.291223] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5032.297578] Call Trace: [ 5032.304679] [ 5032.307255] ? show_regs.cold.9+0x22/0x2f [ 5032.315499] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5032.322842] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5032.331466] ? watchdog+0x40/0x40 [ 5032.333883] ? hrtimer_interrupt+0x128/0x2c0 [ 5032.337935] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5032.346919] ? apic_timer_interrupt+0xf/0x20 [ 5032.348562] [ 5032.349350] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5032.353352] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5032.359240] do_raw_spin_lock+0x177/0x190 [ 5032.361670] _raw_spin_lock+0x21/0x30 [ 5032.363269] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5032.365915] ? do_raw_spin_unlock+0x75/0x190 [ 5032.368254] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5032.370941] ? do_wait_intr+0xf0/0xf0 [ 5032.372695] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5032.375643] mdc_reint+0x2d/0x110 [mdc] [ 5032.377627] mdc_unlink+0x658/0x870 [mdc] [ 5032.379788] lmv_unlink+0x34d/0xbd0 [lmv] [ 5032.382423] ll_unlink+0x24d/0x7f0 [lustre] [ 5032.384657] vfs_unlink+0x202/0x330 [ 5032.388401] do_unlinkat+0x2fa/0x460 [ 5032.394405] __x64_sys_unlink+0x37/0x50 [ 5032.396876] do_syscall_64+0xc1/0x440 [ 5032.399660] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5032.402312] RIP: 0033:0x7f83e5f5c10b [ 5032.408270] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5032.422597] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5032.430872] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5032.438128] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5032.445096] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5032.448883] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5032.455593] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5036.056111] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 5036.062142] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 5036.070879] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5036.071378] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5036.071385] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5036.071388] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5036.071402] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5036.071406] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5036.071410] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5036.071413] RDX: 00000000000078c1 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5036.071415] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5036.071417] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5036.071419] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5036.071422] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5036.071425] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5036.071427] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5036.071437] Call Trace: [ 5036.071442] [ 5036.071448] ? show_regs.cold.9+0x22/0x2f [ 5036.071457] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5036.071463] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5036.071468] ? watchdog+0x40/0x40 [ 5036.071473] ? hrtimer_interrupt+0x128/0x2c0 [ 5036.071477] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5036.071486] ? apic_timer_interrupt+0xf/0x20 [ 5036.071490] [ 5036.071494] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5036.071499] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5036.071504] do_raw_spin_lock+0x177/0x190 [ 5036.071509] _raw_spin_lock+0x21/0x30 [ 5036.071515] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5036.072522] obd_mod_free+0x46/0x140 [ptlrpc] [ 5036.082249] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) [ 5036.115779] mdc_commit_open+0x91/0xb0 [mdc] [ 5036.115829] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5036.116405] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5036.116602] after_reply+0xe2c/0x1780 [ptlrpc] [ 5036.116896] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5036.117429] ? __next_timer_interrupt+0x160/0x160 [ 5036.117439] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5036.118620] ? do_wait_intr+0xf0/0xf0 [ 5036.118630] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5036.118831] kthread+0x1d1/0x200 [ 5036.118837] ? set_kthread_struct+0x70/0x70 [ 5036.118841] ret_from_fork+0x1f/0x30 [ 5036.302006] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5036.336979] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5036.343783] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5036.350466] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5036.356246] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5036.360268] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5036.369476] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5036.376408] RAX: 0000000000007e2a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5036.382988] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5036.390527] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5036.393198] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5036.396716] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5036.402579] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5036.413764] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5036.418377] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5036.428419] Call Trace: [ 5036.430844] [ 5036.432934] ? show_regs.cold.9+0x22/0x2f [ 5036.439489] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5036.449274] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5036.450969] ? watchdog+0x40/0x40 [ 5036.458550] ? hrtimer_interrupt+0x128/0x2c0 [ 5036.461964] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5036.471308] ? apic_timer_interrupt+0xf/0x20 [ 5036.472807] [ 5036.482670] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5036.488711] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5036.493502] do_raw_spin_lock+0x177/0x190 [ 5036.499083] _raw_spin_lock+0x21/0x30 [ 5036.500703] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5036.503955] process_one_work+0x2c8/0x700 [ 5036.505607] worker_thread+0x296/0x6e0 [ 5036.511597] ? rescuer_thread+0x570/0x570 [ 5036.513132] kthread+0x1d1/0x200 [ 5036.514963] ? set_kthread_struct+0x70/0x70 [ 5036.516578] ret_from_fork+0x1f/0x30 [ 5042.144282] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2528s! [ 5042.148966] Showing busy workqueues and worker pools: [ 5042.152383] workqueue events: flags=0x0 [ 5042.154445] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5042.159834] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5042.163519] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5042.167447] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5042.172127] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5042.174748] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5042.177675] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5042.181474] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5042.187075] workqueue events_long: flags=0x0 [ 5042.189802] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5042.193003] pending: lu_objects_flush [obdclass] [ 5042.195228] workqueue events_unbound: flags=0x2 [ 5042.197070] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5042.199700] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5042.202272] workqueue events_power_efficient: flags=0x80 [ 5042.204353] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5042.207223] pending: neigh_periodic_work, do_cache_clean [ 5042.209746] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5042.213434] pending: neigh_periodic_work, check_lifetime [ 5042.216968] workqueue mm_percpu_wq: flags=0x8 [ 5042.218495] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5042.221408] pending: vmstat_update, lru_add_drain_per_cpu [ 5042.224635] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5042.229071] pending: vmstat_update [ 5042.233881] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5042.239923] pending: lru_add_drain_per_cpu BAR(48) [ 5042.243654] workqueue ipv6_addrconf: flags=0x40008 [ 5042.245886] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5042.248619] pending: addrconf_verify_work [ 5042.249852] workqueue knbd0-recv: flags=0x1a [ 5042.251182] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5042.254348] in-flight: 650:recv_work [ 5042.255973] workqueue ptlrpc_pinger: flags=0x2 [ 5042.257968] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5042.260302] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5042.264035] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2529s workers=4 idle: 36 466 29757 [ 5042.267766] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5032s workers=2 idle: 652 [ 5048.059065] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 5048.067813] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5048.115595] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5048.120835] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5048.128987] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5048.133490] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5048.143513] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5048.149734] RAX: 0000000000003493 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5048.155554] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5048.159512] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5048.163388] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5048.168367] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5048.173674] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5048.186263] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5048.189273] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5048.193329] Call Trace: [ 5048.194587] [ 5048.195645] ? show_regs.cold.9+0x22/0x2f [ 5048.198287] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5048.199696] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5048.203650] ? watchdog+0x40/0x40 [ 5048.207530] ? hrtimer_interrupt+0x128/0x2c0 [ 5048.209531] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5048.213014] ? apic_timer_interrupt+0xf/0x20 [ 5048.214107] [ 5048.214554] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5048.221350] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5048.229257] do_raw_spin_lock+0x177/0x190 [ 5048.231683] _raw_spin_lock+0x21/0x30 [ 5048.234441] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5048.237545] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5048.240417] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5048.245269] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5048.248504] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5048.251496] ? woken_wake_function+0x30/0x30 [ 5048.253232] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5048.255606] kthread+0x1d1/0x200 [ 5048.257326] ? set_kthread_struct+0x70/0x70 [ 5048.259339] ret_from_fork+0x1f/0x30 [ 5060.062401] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5060.073716] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5060.132653] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5060.140146] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5060.151824] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5060.159624] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5060.182723] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5060.190271] RAX: 0000000000007088 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5060.197000] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5060.203579] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5060.207365] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5060.214985] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5060.222330] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5060.229810] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5060.237629] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5060.244159] Call Trace: [ 5060.246464] [ 5060.248333] ? show_regs.cold.9+0x22/0x2f [ 5060.251492] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5060.257301] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5060.261402] ? watchdog+0x40/0x40 [ 5060.265031] ? hrtimer_interrupt+0x128/0x2c0 [ 5060.269972] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5060.273257] ? apic_timer_interrupt+0xf/0x20 [ 5060.278162] [ 5060.280231] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5060.284560] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5060.289685] do_raw_spin_lock+0x177/0x190 [ 5060.294389] _raw_spin_lock+0x21/0x30 [ 5060.298222] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5060.305367] ? do_raw_spin_unlock+0x75/0x190 [ 5060.308363] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5060.313365] ? do_wait_intr+0xf0/0xf0 [ 5060.316667] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5060.322602] mdc_reint+0x2d/0x110 [mdc] [ 5060.326830] mdc_unlink+0x658/0x870 [mdc] [ 5060.331429] lmv_unlink+0x34d/0xbd0 [lmv] [ 5060.335968] ll_unlink+0x24d/0x7f0 [lustre] [ 5060.339514] vfs_unlink+0x202/0x330 [ 5060.340906] do_unlinkat+0x2fa/0x460 [ 5060.343500] __x64_sys_unlink+0x37/0x50 [ 5060.348959] do_syscall_64+0xc1/0x440 [ 5060.351626] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5060.358175] RIP: 0033:0x7f83e5f5c10b [ 5060.360603] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5060.381124] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5060.387711] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5060.395036] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5060.400811] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5060.405803] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5060.412397] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5064.060022] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 5064.064627] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5064.107746] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5064.120617] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5064.126016] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5064.129179] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5064.131383] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5064.145608] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5064.150907] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5064.153854] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5064.157466] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5064.164884] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5064.172385] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5064.177583] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5064.181915] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5064.184020] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5064.186704] Call Trace: [ 5064.187703] [ 5064.188450] ? show_regs.cold.9+0x22/0x2f [ 5064.189914] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5064.191941] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5064.193711] ? watchdog+0x40/0x40 [ 5064.194910] ? hrtimer_interrupt+0x128/0x2c0 [ 5064.196539] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5064.198065] ? apic_timer_interrupt+0xf/0x20 [ 5064.199500] [ 5064.200342] ? native_safe_halt+0x2e/0x50 [ 5064.201710] kvm_wait+0xcd/0xe0 [ 5064.202893] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5064.204918] do_raw_spin_lock+0x177/0x190 [ 5064.206599] _raw_spin_lock+0x21/0x30 [ 5064.209899] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5064.214012] process_one_work+0x2c8/0x700 [ 5064.215681] worker_thread+0x296/0x6e0 [ 5064.217315] ? rescuer_thread+0x570/0x570 [ 5064.218900] kthread+0x1d1/0x200 [ 5064.220291] ? set_kthread_struct+0x70/0x70 [ 5064.221939] ret_from_fork+0x1f/0x30 [ 5068.056091] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 5068.060095] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5068.086835] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5068.093523] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5068.098379] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5068.100570] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5068.116731] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5068.120923] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5068.124396] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5068.127484] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5068.133696] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5068.137324] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5068.140354] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5068.144682] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5068.147717] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5068.153219] Call Trace: [ 5068.154829] [ 5068.155928] ? show_regs.cold.9+0x22/0x2f [ 5068.157716] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5068.161219] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5068.163403] ? watchdog+0x40/0x40 [ 5068.165262] ? hrtimer_interrupt+0x128/0x2c0 [ 5068.167384] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5068.169706] ? apic_timer_interrupt+0xf/0x20 [ 5068.171507] [ 5068.172485] ? native_safe_halt+0x2e/0x50 [ 5068.175292] kvm_wait+0xcd/0xe0 [ 5068.177470] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5068.181521] do_raw_spin_lock+0x177/0x190 [ 5068.183294] _raw_spin_lock+0x21/0x30 [ 5068.184747] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5068.187258] obd_mod_free+0x46/0x140 [ptlrpc] [ 5068.189138] mdc_commit_open+0x91/0xb0 [mdc] [ 5068.191332] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5068.194211] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5068.196852] after_reply+0xe2c/0x1780 [ptlrpc] [ 5068.199461] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5068.201481] ? __next_timer_interrupt+0x160/0x160 [ 5068.204190] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5068.207610] ? do_wait_intr+0xf0/0xf0 [ 5068.209256] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5068.212297] kthread+0x1d1/0x200 [ 5068.213492] ? set_kthread_struct+0x70/0x70 [ 5068.215516] ret_from_fork+0x1f/0x30 [ 5072.353078] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2559s! [ 5072.356371] Showing busy workqueues and worker pools: [ 5072.358086] workqueue events: flags=0x0 [ 5072.359522] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5072.361901] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5072.366561] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5072.369862] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5072.371937] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5072.375517] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5072.379578] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5072.389473] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5072.397202] workqueue events_long: flags=0x0 [ 5072.400181] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5072.403150] pending: lu_objects_flush [obdclass] [ 5072.408513] workqueue events_unbound: flags=0x2 [ 5072.411539] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5072.415665] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5072.421502] workqueue events_power_efficient: flags=0x80 [ 5072.425423] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5072.434481] pending: neigh_periodic_work, do_cache_clean [ 5072.437633] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5072.441826] pending: neigh_periodic_work, check_lifetime [ 5072.446219] workqueue mm_percpu_wq: flags=0x8 [ 5072.448191] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5072.451417] pending: vmstat_update, lru_add_drain_per_cpu [ 5072.454223] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5072.457116] pending: vmstat_update [ 5072.458501] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5072.460921] pending: lru_add_drain_per_cpu BAR(48) [ 5072.463282] workqueue ipv6_addrconf: flags=0x40008 [ 5072.464903] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5072.467200] pending: addrconf_verify_work [ 5072.468740] workqueue knbd0-recv: flags=0x1a [ 5072.470402] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5072.472938] in-flight: 650:recv_work [ 5072.474597] workqueue ptlrpc_pinger: flags=0x2 [ 5072.476164] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5072.477811] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5072.486645] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2559s workers=4 idle: 36 466 29757 [ 5072.492351] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5062s workers=2 idle: 652 [ 5076.059036] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 5076.062572] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5076.086244] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5076.091131] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5076.096064] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5076.098285] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5076.105854] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5076.110773] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5076.114380] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5076.117498] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5076.120245] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5076.122820] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5076.125508] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5076.128545] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5076.130954] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5076.133958] Call Trace: [ 5076.135039] [ 5076.135672] ? show_regs.cold.9+0x22/0x2f [ 5076.137296] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5076.139117] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5076.140844] ? watchdog+0x40/0x40 [ 5076.142405] ? hrtimer_interrupt+0x128/0x2c0 [ 5076.144232] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5076.145823] ? apic_timer_interrupt+0xf/0x20 [ 5076.147518] [ 5076.148381] ? native_safe_halt+0x2e/0x50 [ 5076.149641] kvm_wait+0xcd/0xe0 [ 5076.150899] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5076.152875] do_raw_spin_lock+0x177/0x190 [ 5076.154456] _raw_spin_lock+0x21/0x30 [ 5076.155946] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5076.158972] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5076.161306] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5076.163815] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5076.166540] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5076.168905] ? woken_wake_function+0x30/0x30 [ 5076.170634] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5076.173674] kthread+0x1d1/0x200 [ 5076.175873] ? set_kthread_struct+0x70/0x70 [ 5076.177894] ret_from_fork+0x1f/0x30 [ 5088.062246] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5088.074168] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5088.106384] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5088.118364] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5088.122998] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5088.125932] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5088.139051] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5088.146456] RAX: 0000000000003b85 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5088.149601] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5088.152946] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5088.157373] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5088.161222] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5088.164337] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5088.169038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5088.171727] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5088.174600] Call Trace: [ 5088.175696] [ 5088.176581] ? show_regs.cold.9+0x22/0x2f [ 5088.178337] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5088.180401] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5088.182390] ? watchdog+0x40/0x40 [ 5088.183982] ? hrtimer_interrupt+0x128/0x2c0 [ 5088.186384] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5088.189080] ? apic_timer_interrupt+0xf/0x20 [ 5088.193531] [ 5088.194569] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5088.197030] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5088.199327] do_raw_spin_lock+0x177/0x190 [ 5088.200976] _raw_spin_lock+0x21/0x30 [ 5088.202331] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5088.206255] ? do_raw_spin_unlock+0x75/0x190 [ 5088.212495] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5088.219757] ? do_wait_intr+0xf0/0xf0 [ 5088.223327] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5088.230244] mdc_reint+0x2d/0x110 [mdc] [ 5088.233136] mdc_unlink+0x658/0x870 [mdc] [ 5088.235107] lmv_unlink+0x34d/0xbd0 [lmv] [ 5088.237219] ll_unlink+0x24d/0x7f0 [lustre] [ 5088.240004] vfs_unlink+0x202/0x330 [ 5088.241782] do_unlinkat+0x2fa/0x460 [ 5088.244221] __x64_sys_unlink+0x37/0x50 [ 5088.245573] do_syscall_64+0xc1/0x440 [ 5088.246678] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5088.249296] RIP: 0033:0x7f83e5f5c10b [ 5088.250551] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5088.258284] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5088.261313] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5088.265199] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5088.267882] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5088.271349] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5088.273923] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5092.064274] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 5092.068572] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5092.107203] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5092.124116] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5092.134841] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5092.140238] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5092.144164] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5092.155217] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5092.161741] RAX: 0000000000000de6 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5092.167217] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5092.176862] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5092.183308] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5092.188896] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5092.196073] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5092.203667] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5092.210136] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5092.219032] Call Trace: [ 5092.222004] [ 5092.222959] ? show_regs.cold.9+0x22/0x2f [ 5092.226727] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5092.234225] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5092.239877] ? watchdog+0x40/0x40 [ 5092.241430] ? hrtimer_interrupt+0x128/0x2c0 [ 5092.249201] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5092.254896] ? apic_timer_interrupt+0xf/0x20 [ 5092.260433] [ 5092.262184] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5092.267698] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5092.271939] do_raw_spin_lock+0x177/0x190 [ 5092.274977] _raw_spin_lock+0x21/0x30 [ 5092.277224] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5092.280341] process_one_work+0x2c8/0x700 [ 5092.282565] worker_thread+0x296/0x6e0 [ 5092.284710] ? rescuer_thread+0x570/0x570 [ 5092.286802] kthread+0x1d1/0x200 [ 5092.293039] ? set_kthread_struct+0x70/0x70 [ 5092.296859] ret_from_fork+0x1f/0x30 [ 5100.056826] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 5100.060246] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5100.085090] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5100.090556] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5100.094987] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5100.097762] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5100.106804] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5100.110420] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5100.113975] RDX: 000000000000615a RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5100.116805] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5100.120556] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5100.124880] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5100.127868] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5100.131283] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5100.133725] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5100.136647] Call Trace: [ 5100.137764] [ 5100.138509] ? show_regs.cold.9+0x22/0x2f [ 5100.140636] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5100.142525] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5100.144490] ? watchdog+0x40/0x40 [ 5100.146006] ? hrtimer_interrupt+0x128/0x2c0 [ 5100.148131] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5100.150431] ? apic_timer_interrupt+0xf/0x20 [ 5100.152433] [ 5100.153831] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5100.156182] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5100.158455] do_raw_spin_lock+0x177/0x190 [ 5100.160137] _raw_spin_lock+0x21/0x30 [ 5100.161811] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5100.164796] obd_mod_free+0x46/0x140 [ptlrpc] [ 5100.167242] mdc_commit_open+0x91/0xb0 [mdc] [ 5100.169296] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5100.171965] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5100.174506] after_reply+0xe2c/0x1780 [ptlrpc] [ 5100.177033] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5100.179628] ? __next_timer_interrupt+0x160/0x160 [ 5100.182211] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5100.184254] ? do_wait_intr+0xf0/0xf0 [ 5100.186030] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5100.189087] kthread+0x1d1/0x200 [ 5100.190293] ? set_kthread_struct+0x70/0x70 [ 5100.192151] ret_from_fork+0x1f/0x30 [ 5101.028020] rcu: INFO: rcu_sched self-detected stall on CPU [ 5101.030132] rcu: 0-....: (1947097 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 5101.034004] (t=3120074 jiffies g=262441 q=58054) [ 5101.035593] rcu: rcu_sched kthread starved for 2589263 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 5101.039573] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5101.043009] rcu: RCU grace-period kthread stack dump: [ 5101.045067] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5101.048720] Call Trace: [ 5101.049649] __schedule+0x351/0xcb0 [ 5101.051059] schedule+0xc0/0x180 [ 5101.052243] schedule_timeout+0xb4/0x190 [ 5101.053739] ? do_raw_spin_unlock+0x75/0x190 [ 5101.055151] ? __next_timer_interrupt+0x160/0x160 [ 5101.057018] ? prepare_to_swait_event+0xf3/0x140 [ 5101.058958] rcu_gp_kthread+0x8dd/0x15b0 [ 5101.060229] ? force_qs_rnp+0x2c0/0x2c0 [ 5101.061805] kthread+0x1d1/0x200 [ 5101.062990] ? set_kthread_struct+0x70/0x70 [ 5101.064918] ret_from_fork+0x1f/0x30 [ 5101.066145] rcu: Stack dump where RCU GP kthread last ran: [ 5101.068135] Sending NMI from CPU 0 to CPUs 1: [ 5101.069889] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 5101.072246] NMI backtrace for cpu 0 [ 5101.073926] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5101.078559] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5101.082598] Call Trace: [ 5101.083587] [ 5101.084559] ? dump_stack+0xbb/0x10e [ 5101.085993] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5101.088593] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5101.090613] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5101.093095] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5101.095119] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5101.097230] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5101.099134] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5101.101285] ? update_process_times+0x6d/0xc0 [ 5101.103226] ? tick_sched_handle.isra.18+0x27/0x90 [ 5101.105153] ? tick_sched_timer+0x53/0xb0 [ 5101.106962] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5101.108983] ? tick_sched_handle.isra.18+0x90/0x90 [ 5101.110823] ? hrtimer_interrupt+0x128/0x2c0 [ 5101.112896] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5101.115045] ? apic_timer_interrupt+0xf/0x20 [ 5101.116908] [ 5101.117721] ? native_safe_halt+0x2e/0x50 [ 5101.119321] ? kvm_wait+0xcd/0xe0 [ 5101.121101] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5101.123412] ? do_raw_spin_lock+0x177/0x190 [ 5101.125086] ? _raw_spin_lock+0x21/0x30 [ 5101.126874] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5101.129585] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5101.133129] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5101.135260] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5101.137724] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5101.140950] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5101.143072] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5101.145554] ? __next_timer_interrupt+0x160/0x160 [ 5101.147520] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5101.149537] ? do_wait_intr+0xf0/0xf0 [ 5101.150891] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5101.153063] ? kthread+0x1d1/0x200 [ 5101.154526] ? set_kthread_struct+0x70/0x70 [ 5101.156169] ? ret_from_fork+0x1f/0x30 [ 5102.561064] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2589s! [ 5102.563599] Showing busy workqueues and worker pools: [ 5102.565877] workqueue events: flags=0x0 [ 5102.567189] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5102.569685] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5102.572721] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5102.575161] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5102.578106] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5102.580317] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5102.583439] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5102.585742] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5102.588831] workqueue events_long: flags=0x0 [ 5102.590368] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5102.592494] pending: lu_objects_flush [obdclass] [ 5102.594647] workqueue events_unbound: flags=0x2 [ 5102.596182] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5102.598442] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5102.600569] workqueue events_power_efficient: flags=0x80 [ 5102.602620] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5102.604967] pending: neigh_periodic_work, do_cache_clean [ 5102.606991] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5102.609467] pending: neigh_periodic_work, check_lifetime [ 5102.611480] workqueue mm_percpu_wq: flags=0x8 [ 5102.613220] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5102.615483] pending: vmstat_update, lru_add_drain_per_cpu [ 5102.616894] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5102.618952] pending: vmstat_update [ 5102.620376] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5102.622799] pending: lru_add_drain_per_cpu BAR(48) [ 5102.624848] workqueue ipv6_addrconf: flags=0x40008 [ 5102.626225] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5102.628471] pending: addrconf_verify_work [ 5102.629822] workqueue knbd0-recv: flags=0x1a [ 5102.631144] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5102.633243] in-flight: 650:recv_work [ 5102.634437] workqueue ptlrpc_pinger: flags=0x2 [ 5102.636128] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5102.638559] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5102.640704] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2589s workers=4 idle: 36 466 29757 [ 5102.643856] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5092s workers=2 idle: 652 [ 5116.062014] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5116.064654] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5116.079949] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5116.083524] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5116.086680] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5116.088352] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5116.099591] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5116.102198] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5116.104668] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5116.107184] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5116.109592] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5116.112320] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5116.114867] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5116.117862] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5116.119838] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5116.122266] Call Trace: [ 5116.123130] [ 5116.124262] ? show_regs.cold.9+0x22/0x2f [ 5116.126147] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5116.129011] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5116.132695] ? watchdog+0x40/0x40 [ 5116.134639] ? hrtimer_interrupt+0x128/0x2c0 [ 5116.139488] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5116.141227] ? apic_timer_interrupt+0xf/0x20 [ 5116.142661] [ 5116.143385] ? native_safe_halt+0x2e/0x50 [ 5116.144693] kvm_wait+0xcd/0xe0 [ 5116.145708] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5116.147371] do_raw_spin_lock+0x177/0x190 [ 5116.148742] _raw_spin_lock+0x21/0x30 [ 5116.149896] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5116.152264] ? do_raw_spin_unlock+0x75/0x190 [ 5116.153838] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5116.155766] ? do_wait_intr+0xf0/0xf0 [ 5116.156894] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5116.159098] mdc_reint+0x2d/0x110 [mdc] [ 5116.160376] mdc_unlink+0x658/0x870 [mdc] [ 5116.161652] lmv_unlink+0x34d/0xbd0 [lmv] [ 5116.163040] ll_unlink+0x24d/0x7f0 [lustre] [ 5116.164912] vfs_unlink+0x202/0x330 [ 5116.166063] do_unlinkat+0x2fa/0x460 [ 5116.167443] __x64_sys_unlink+0x37/0x50 [ 5116.168695] do_syscall_64+0xc1/0x440 [ 5116.170018] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5116.171646] RIP: 0033:0x7f83e5f5c10b [ 5116.172905] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5116.179112] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5116.181701] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5116.184180] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5116.186686] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5116.189767] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5116.192492] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5120.061082] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5120.066041] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5120.113154] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5120.124810] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5120.136617] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5120.139764] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5120.148040] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5120.161251] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5120.165648] RAX: 0000000000005612 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5120.169145] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5120.173891] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5120.175491] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5120.179140] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5120.181810] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5120.184477] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5120.187470] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5120.193092] Call Trace: [ 5120.195357] [ 5120.195845] ? show_regs.cold.9+0x22/0x2f [ 5120.197793] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5120.198963] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5120.200189] ? watchdog+0x40/0x40 [ 5120.201999] ? hrtimer_interrupt+0x128/0x2c0 [ 5120.205560] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5120.207103] ? apic_timer_interrupt+0xf/0x20 [ 5120.209297] [ 5120.209919] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5120.212545] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5120.215895] do_raw_spin_lock+0x177/0x190 [ 5120.217751] _raw_spin_lock+0x21/0x30 [ 5120.219604] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5120.222216] process_one_work+0x2c8/0x700 [ 5120.225825] worker_thread+0x296/0x6e0 [ 5120.227635] ? rescuer_thread+0x570/0x570 [ 5120.232359] kthread+0x1d1/0x200 [ 5120.235594] ? set_kthread_struct+0x70/0x70 [ 5120.237664] ret_from_fork+0x1f/0x30 [ 5128.056061] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 5128.059033] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 5128.060295] Modules linked in: [ 5128.065833] Modules linked in: [ 5128.065835] lustre(O) osp(O) [ 5128.067125] lustre(O) [ 5128.068680] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5128.068753] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5128.068760] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5128.068763] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5128.068777] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5128.068780] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5128.068785] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5128.068788] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5128.068790] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5128.068792] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5128.068794] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5128.068798] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5128.068800] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5128.068802] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5128.068812] Call Trace: [ 5128.068817] [ 5128.068823] ? show_regs.cold.9+0x22/0x2f [ 5128.068834] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5128.068841] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5128.068846] ? watchdog+0x40/0x40 [ 5128.068850] ? hrtimer_interrupt+0x128/0x2c0 [ 5128.068854] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5128.068861] ? apic_timer_interrupt+0xf/0x20 [ 5128.068865] [ 5128.068868] ? native_safe_halt+0x2e/0x50 [ 5128.068872] kvm_wait+0xcd/0xe0 [ 5128.068880] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5128.068888] do_raw_spin_lock+0x177/0x190 [ 5128.068892] _raw_spin_lock+0x21/0x30 [ 5128.068898] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5128.069355] obd_mod_free+0x46/0x140 [ptlrpc] [ 5128.069670] mdc_commit_open+0x91/0xb0 [mdc] [ 5128.069709] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5128.069927] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5128.070473] after_reply+0xe2c/0x1780 [ptlrpc] [ 5128.070685] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5128.070890] ? __next_timer_interrupt+0x160/0x160 [ 5128.070900] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5128.071120] ? do_wait_intr+0xf0/0xf0 [ 5128.071129] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5128.072268] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) [ 5128.073349] kthread+0x1d1/0x200 [ 5128.073362] ? set_kthread_struct+0x70/0x70 [ 5128.073366] ret_from_fork+0x1f/0x30 [ 5128.220933] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5128.239278] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5128.248601] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5128.254245] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5128.258109] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5128.271029] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5128.274952] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5128.284341] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5128.293482] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5128.297006] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5128.300698] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5128.310499] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5128.315005] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5128.322035] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5128.326685] Call Trace: [ 5128.327578] [ 5128.328230] ? show_regs.cold.9+0x22/0x2f [ 5128.329141] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5128.334903] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5128.337009] ? watchdog+0x40/0x40 [ 5128.339881] ? hrtimer_interrupt+0x128/0x2c0 [ 5128.345100] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5128.348785] ? apic_timer_interrupt+0xf/0x20 [ 5128.350097] [ 5128.352105] ? native_safe_halt+0x2e/0x50 [ 5128.356085] kvm_wait+0xcd/0xe0 [ 5128.359510] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5128.362349] do_raw_spin_lock+0x177/0x190 [ 5128.364668] _raw_spin_lock+0x21/0x30 [ 5128.366948] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5128.370568] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5128.374953] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5128.379449] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5128.381630] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5128.386760] ? woken_wake_function+0x30/0x30 [ 5128.389786] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5128.392991] kthread+0x1d1/0x200 [ 5128.394019] ? set_kthread_struct+0x70/0x70 [ 5128.395837] ret_from_fork+0x1f/0x30 [ 5132.768141] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2619s! [ 5132.772253] Showing busy workqueues and worker pools: [ 5132.779290] workqueue events: flags=0x0 [ 5132.781414] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5132.787274] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5132.798450] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5132.801653] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5132.809374] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5132.816098] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5132.826531] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5132.834671] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5132.842903] workqueue events_long: flags=0x0 [ 5132.845228] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5132.851271] pending: lu_objects_flush [obdclass] [ 5132.854353] workqueue events_unbound: flags=0x2 [ 5132.857200] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5132.860551] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5132.863364] workqueue events_power_efficient: flags=0x80 [ 5132.866559] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5132.870754] pending: neigh_periodic_work, do_cache_clean [ 5132.874302] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5132.878100] pending: neigh_periodic_work, check_lifetime [ 5132.881645] workqueue mm_percpu_wq: flags=0x8 [ 5132.884077] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5132.887531] pending: vmstat_update, lru_add_drain_per_cpu [ 5132.890607] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5132.894281] pending: vmstat_update [ 5132.896293] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5132.900124] pending: lru_add_drain_per_cpu BAR(48) [ 5132.903149] workqueue ipv6_addrconf: flags=0x40008 [ 5132.905212] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5132.907678] pending: addrconf_verify_work [ 5132.909206] workqueue knbd0-recv: flags=0x1a [ 5132.911292] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5132.915310] in-flight: 650:recv_work [ 5132.917107] workqueue ptlrpc_pinger: flags=0x2 [ 5132.919409] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5132.922145] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5132.926967] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2619s workers=4 idle: 36 466 29757 [ 5132.931689] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5122s workers=2 idle: 652 [ 5144.062100] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5144.065646] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5144.089806] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5144.096281] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5144.100805] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5144.104742] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5144.112694] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5144.116238] RAX: 0000000000005baf RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5144.120565] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5144.124519] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5144.128173] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5144.131866] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5144.136025] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5144.140298] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5144.143166] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5144.147331] Call Trace: [ 5144.148649] [ 5144.149718] ? show_regs.cold.9+0x22/0x2f [ 5144.152121] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5144.154527] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5144.156862] ? watchdog+0x40/0x40 [ 5144.159163] ? hrtimer_interrupt+0x128/0x2c0 [ 5144.161844] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5144.165120] ? apic_timer_interrupt+0xf/0x20 [ 5144.168405] [ 5144.169380] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5144.173102] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5144.175560] do_raw_spin_lock+0x177/0x190 [ 5144.177643] _raw_spin_lock+0x21/0x30 [ 5144.179513] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5144.183194] ? do_raw_spin_unlock+0x75/0x190 [ 5144.185162] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5144.187889] ? do_wait_intr+0xf0/0xf0 [ 5144.189721] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5144.192652] mdc_reint+0x2d/0x110 [mdc] [ 5144.194736] mdc_unlink+0x658/0x870 [mdc] [ 5144.197071] lmv_unlink+0x34d/0xbd0 [lmv] [ 5144.199306] ll_unlink+0x24d/0x7f0 [lustre] [ 5144.201469] vfs_unlink+0x202/0x330 [ 5144.203130] do_unlinkat+0x2fa/0x460 [ 5144.205183] __x64_sys_unlink+0x37/0x50 [ 5144.207165] do_syscall_64+0xc1/0x440 [ 5144.208865] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5144.211686] RIP: 0033:0x7f83e5f5c10b [ 5144.213155] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5144.222538] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5144.226551] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5144.229755] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5144.233032] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5144.236460] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5144.239854] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5148.060029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5148.062695] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5148.090626] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5148.100825] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5148.110773] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5148.119294] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5148.124346] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5148.144811] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5148.159672] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5148.174667] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5148.184731] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5148.196305] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5148.205581] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5148.211971] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5148.224187] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5148.243667] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5148.253752] Call Trace: [ 5148.255919] [ 5148.257692] ? show_regs.cold.9+0x22/0x2f [ 5148.259835] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5148.264214] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5148.268576] ? watchdog+0x40/0x40 [ 5148.269480] ? hrtimer_interrupt+0x128/0x2c0 [ 5148.270764] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5148.272735] ? apic_timer_interrupt+0xf/0x20 [ 5148.274295] [ 5148.274817] ? native_safe_halt+0x2e/0x50 [ 5148.276370] kvm_wait+0xcd/0xe0 [ 5148.277642] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5148.279159] do_raw_spin_lock+0x177/0x190 [ 5148.280319] _raw_spin_lock+0x21/0x30 [ 5148.281336] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5148.283044] process_one_work+0x2c8/0x700 [ 5148.284096] worker_thread+0x296/0x6e0 [ 5148.285025] ? rescuer_thread+0x570/0x570 [ 5148.286219] kthread+0x1d1/0x200 [ 5148.287061] ? set_kthread_struct+0x70/0x70 [ 5148.289396] ret_from_fork+0x1f/0x30 [ 5156.059019] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 5156.061399] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5156.076918] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5156.080672] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5156.083816] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5156.085796] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5156.093626] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5156.097064] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5156.099829] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5156.102337] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5156.104914] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5156.107604] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5156.110495] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5156.113608] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5156.115622] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5156.124595] Call Trace: [ 5156.128435] [ 5156.130033] ? show_regs.cold.9+0x22/0x2f [ 5156.133150] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5156.136115] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5156.138232] ? watchdog+0x40/0x40 [ 5156.142440] ? hrtimer_interrupt+0x128/0x2c0 [ 5156.145751] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5156.148119] ? apic_timer_interrupt+0xf/0x20 [ 5156.149889] [ 5156.150735] ? native_safe_halt+0x2e/0x50 [ 5156.152376] kvm_wait+0xcd/0xe0 [ 5156.154049] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5156.156140] do_raw_spin_lock+0x177/0x190 [ 5156.157670] _raw_spin_lock+0x21/0x30 [ 5156.159754] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5156.167670] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5156.174697] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5156.178476] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5156.181192] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5156.187417] ? woken_wake_function+0x30/0x30 [ 5156.189543] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5156.194106] kthread+0x1d1/0x200 [ 5156.196545] ? set_kthread_struct+0x70/0x70 [ 5156.198686] ret_from_fork+0x1f/0x30 [ 5160.056021] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 5160.059134] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5160.080963] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5160.094071] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5160.102773] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5160.108779] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5160.120138] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5160.133698] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5160.140561] RDX: 00000000000061d3 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5160.149109] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5160.159937] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5160.166921] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5160.171745] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5160.181007] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5160.188076] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5160.194114] Call Trace: [ 5160.195928] [ 5160.196844] ? show_regs.cold.9+0x22/0x2f [ 5160.199466] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5160.205288] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5160.208008] ? watchdog+0x40/0x40 [ 5160.210284] ? hrtimer_interrupt+0x128/0x2c0 [ 5160.213266] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5160.217050] ? apic_timer_interrupt+0xf/0x20 [ 5160.220165] [ 5160.222262] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5160.230766] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5160.238187] do_raw_spin_lock+0x177/0x190 [ 5160.244258] _raw_spin_lock+0x21/0x30 [ 5160.248365] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5160.251715] obd_mod_free+0x46/0x140 [ptlrpc] [ 5160.255063] mdc_commit_open+0x91/0xb0 [mdc] [ 5160.258120] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5160.261698] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5160.265304] after_reply+0xe2c/0x1780 [ptlrpc] [ 5160.272905] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5160.277496] ? __next_timer_interrupt+0x160/0x160 [ 5160.280652] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5160.286329] ? do_wait_intr+0xf0/0xf0 [ 5160.288292] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5160.291221] kthread+0x1d1/0x200 [ 5160.292724] ? set_kthread_struct+0x70/0x70 [ 5160.294254] ret_from_fork+0x1f/0x30 [ 5162.982333] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2649s! [ 5162.989181] Showing busy workqueues and worker pools: [ 5162.996015] workqueue events: flags=0x0 [ 5162.999544] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5163.002932] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5163.010282] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5163.017421] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5163.024240] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5163.029464] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5163.033532] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5163.041598] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5163.049172] workqueue events_long: flags=0x0 [ 5163.054046] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5163.060364] pending: lu_objects_flush [obdclass] [ 5163.062890] workqueue events_unbound: flags=0x2 [ 5163.068733] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5163.074043] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5163.080883] workqueue events_power_efficient: flags=0x80 [ 5163.083926] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5163.090318] pending: neigh_periodic_work, do_cache_clean [ 5163.095990] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5163.102749] pending: neigh_periodic_work, check_lifetime [ 5163.107815] workqueue mm_percpu_wq: flags=0x8 [ 5163.110598] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5163.115388] pending: vmstat_update, lru_add_drain_per_cpu [ 5163.118348] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5163.124929] pending: vmstat_update [ 5163.127950] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5163.132596] pending: lru_add_drain_per_cpu BAR(48) [ 5163.137138] workqueue ipv6_addrconf: flags=0x40008 [ 5163.141679] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5163.143547] pending: addrconf_verify_work [ 5163.146165] workqueue knbd0-recv: flags=0x1a [ 5163.149570] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5163.153402] in-flight: 650:recv_work [ 5163.154991] workqueue ptlrpc_pinger: flags=0x2 [ 5163.157018] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5163.160318] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5163.163545] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2649s workers=4 idle: 36 466 29757 [ 5163.166919] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5153s workers=2 idle: 652 [ 5172.062017] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5172.066188] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5172.090473] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5172.096360] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5172.100416] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5172.103307] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5172.114040] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5172.116644] RAX: 0000000000005ec5 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5172.119663] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5172.126535] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5172.129481] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5172.134364] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5172.137344] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5172.145212] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5172.151556] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5172.159127] Call Trace: [ 5172.161245] [ 5172.162577] ? show_regs.cold.9+0x22/0x2f [ 5172.164242] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5172.169119] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5172.171314] ? watchdog+0x40/0x40 [ 5172.174315] ? hrtimer_interrupt+0x128/0x2c0 [ 5172.178839] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5172.181513] ? apic_timer_interrupt+0xf/0x20 [ 5172.185964] [ 5172.186553] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5172.190351] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5172.194122] do_raw_spin_lock+0x177/0x190 [ 5172.195487] _raw_spin_lock+0x21/0x30 [ 5172.198268] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5172.200421] ? do_raw_spin_unlock+0x75/0x190 [ 5172.202942] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5172.205215] ? do_wait_intr+0xf0/0xf0 [ 5172.206514] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5172.208965] mdc_reint+0x2d/0x110 [mdc] [ 5172.210875] mdc_unlink+0x658/0x870 [mdc] [ 5172.212484] lmv_unlink+0x34d/0xbd0 [lmv] [ 5172.214105] ll_unlink+0x24d/0x7f0 [lustre] [ 5172.216125] vfs_unlink+0x202/0x330 [ 5172.217246] do_unlinkat+0x2fa/0x460 [ 5172.218466] __x64_sys_unlink+0x37/0x50 [ 5172.220396] do_syscall_64+0xc1/0x440 [ 5172.221755] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5172.223405] RIP: 0033:0x7f83e5f5c10b [ 5172.224818] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5172.231099] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5172.233436] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5172.235831] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5172.238098] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5172.240489] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5172.242663] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5176.060034] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5176.062999] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5176.106561] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5176.112432] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5176.116726] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5176.122262] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5176.124998] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5176.130865] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5176.132934] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5176.142766] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5176.155065] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5176.168176] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5176.177433] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5176.182556] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5176.188322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5176.191002] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5176.199931] Call Trace: [ 5176.200824] [ 5176.201511] ? show_regs.cold.9+0x22/0x2f [ 5176.208155] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5176.210841] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5176.217375] ? watchdog+0x40/0x40 [ 5176.219064] ? hrtimer_interrupt+0x128/0x2c0 [ 5176.222406] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5176.225097] ? apic_timer_interrupt+0xf/0x20 [ 5176.227122] [ 5176.228262] ? native_safe_halt+0x2e/0x50 [ 5176.230558] kvm_wait+0xcd/0xe0 [ 5176.231619] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5176.237098] do_raw_spin_lock+0x177/0x190 [ 5176.239132] _raw_spin_lock+0x21/0x30 [ 5176.240735] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5176.243907] process_one_work+0x2c8/0x700 [ 5176.246898] worker_thread+0x296/0x6e0 [ 5176.248645] ? rescuer_thread+0x570/0x570 [ 5176.251358] kthread+0x1d1/0x200 [ 5176.253330] ? set_kthread_struct+0x70/0x70 [ 5176.255329] ret_from_fork+0x1f/0x30 [ 5184.059418] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 5184.063201] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5184.084660] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5184.092564] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5184.095293] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5184.102258] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5184.110087] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5184.114032] RAX: 000000000000758b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5184.118389] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5184.121515] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5184.124560] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5184.127289] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5184.129806] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5184.134128] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5184.137521] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5184.140823] Call Trace: [ 5184.142126] [ 5184.143006] ? show_regs.cold.9+0x22/0x2f [ 5184.144624] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5184.147243] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5184.148991] ? watchdog+0x40/0x40 [ 5184.156279] ? hrtimer_interrupt+0x128/0x2c0 [ 5184.159690] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5184.162100] ? apic_timer_interrupt+0xf/0x20 [ 5184.163389] [ 5184.164552] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5184.169353] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5184.173319] do_raw_spin_lock+0x177/0x190 [ 5184.179642] _raw_spin_lock+0x21/0x30 [ 5184.181497] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5184.187205] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5184.189131] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5184.191842] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5184.193541] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5184.196303] ? woken_wake_function+0x30/0x30 [ 5184.200567] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5184.203063] kthread+0x1d1/0x200 [ 5184.206675] ? set_kthread_struct+0x70/0x70 [ 5184.209067] ret_from_fork+0x1f/0x30 [ 5188.056049] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 5188.060551] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5188.085066] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5188.089289] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5188.092050] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5188.093884] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5188.100352] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5188.102974] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5188.105500] RDX: 0000000000000bff RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5188.107885] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5188.110052] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5188.112068] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5188.114199] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5188.116590] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5188.118518] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5188.121159] Call Trace: [ 5188.121931] [ 5188.122489] ? show_regs.cold.9+0x22/0x2f [ 5188.123714] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5188.125227] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5188.127002] ? watchdog+0x40/0x40 [ 5188.128087] ? hrtimer_interrupt+0x128/0x2c0 [ 5188.129431] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5188.130951] ? apic_timer_interrupt+0xf/0x20 [ 5188.132290] [ 5188.132957] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5188.134557] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5188.136195] do_raw_spin_lock+0x177/0x190 [ 5188.137434] _raw_spin_lock+0x21/0x30 [ 5188.138672] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5188.140401] obd_mod_free+0x46/0x140 [ptlrpc] [ 5188.142070] mdc_commit_open+0x91/0xb0 [mdc] [ 5188.143338] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5188.145760] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5188.147866] after_reply+0xe2c/0x1780 [ptlrpc] [ 5188.149955] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5188.151974] ? __next_timer_interrupt+0x160/0x160 [ 5188.153508] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5188.155293] ? do_wait_intr+0xf0/0xf0 [ 5188.156448] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5188.158389] kthread+0x1d1/0x200 [ 5188.159495] ? set_kthread_struct+0x70/0x70 [ 5188.160772] ret_from_fork+0x1f/0x30 [ 5193.185043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2680s! [ 5193.192437] Showing busy workqueues and worker pools: [ 5193.200613] workqueue events: flags=0x0 [ 5193.210927] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5193.216759] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5193.232692] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5193.237457] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5193.242627] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5193.248482] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5193.254480] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5193.258159] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5193.262749] workqueue events_long: flags=0x0 [ 5193.264818] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5193.267687] pending: lu_objects_flush [obdclass] [ 5193.270223] workqueue events_unbound: flags=0x2 [ 5193.272089] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5193.274457] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5193.277263] workqueue events_power_efficient: flags=0x80 [ 5193.279367] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5193.282183] pending: neigh_periodic_work, do_cache_clean [ 5193.284990] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5193.287829] pending: neigh_periodic_work, check_lifetime [ 5193.290500] workqueue mm_percpu_wq: flags=0x8 [ 5193.292471] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5193.296032] pending: vmstat_update, lru_add_drain_per_cpu [ 5193.297610] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5193.300127] pending: vmstat_update [ 5193.301977] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5193.304492] pending: lru_add_drain_per_cpu BAR(48) [ 5193.306954] workqueue ipv6_addrconf: flags=0x40008 [ 5193.309058] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5193.311582] pending: addrconf_verify_work [ 5193.313280] workqueue knbd0-recv: flags=0x1a [ 5193.314943] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5193.317310] in-flight: 650:recv_work [ 5193.318751] workqueue ptlrpc_pinger: flags=0x2 [ 5193.320256] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5193.322633] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5193.325181] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2680s workers=4 idle: 36 466 29757 [ 5193.327697] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5183s workers=2 idle: 652 [ 5200.062008] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5200.067348] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5200.094645] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5200.100849] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5200.104318] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5200.107369] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5200.114770] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5200.117696] RAX: 0000000000001999 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5200.120251] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5200.123265] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5200.125663] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5200.128024] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5200.130941] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5200.133615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5200.135902] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5200.138483] Call Trace: [ 5200.139189] [ 5200.139968] ? show_regs.cold.9+0x22/0x2f [ 5200.141543] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5200.143699] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5200.145956] ? watchdog+0x40/0x40 [ 5200.147300] ? hrtimer_interrupt+0x128/0x2c0 [ 5200.148966] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5200.150702] ? apic_timer_interrupt+0xf/0x20 [ 5200.152479] [ 5200.153470] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5200.155701] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5200.157795] do_raw_spin_lock+0x177/0x190 [ 5200.159097] _raw_spin_lock+0x21/0x30 [ 5200.160441] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5200.162684] ? do_raw_spin_unlock+0x75/0x190 [ 5200.164441] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5200.166652] ? do_wait_intr+0xf0/0xf0 [ 5200.168004] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5200.170361] mdc_reint+0x2d/0x110 [mdc] [ 5200.171518] mdc_unlink+0x658/0x870 [mdc] [ 5200.172752] lmv_unlink+0x34d/0xbd0 [lmv] [ 5200.174076] ll_unlink+0x24d/0x7f0 [lustre] [ 5200.175564] vfs_unlink+0x202/0x330 [ 5200.176863] do_unlinkat+0x2fa/0x460 [ 5200.178304] __x64_sys_unlink+0x37/0x50 [ 5200.179444] do_syscall_64+0xc1/0x440 [ 5200.180832] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5200.182211] RIP: 0033:0x7f83e5f5c10b [ 5200.183907] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5200.190427] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5200.192973] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5200.195537] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5200.198178] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5200.200615] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5200.202935] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5204.060171] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5204.062554] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5204.080550] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5204.084522] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5204.093211] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5204.098548] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5204.100520] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5204.115276] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [ 5204.118444] RAX: 0000000000004800 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5204.120967] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5204.123250] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5204.125605] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5204.128316] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5204.132794] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5204.135524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5204.137633] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5204.139547] Call Trace: [ 5204.140219] [ 5204.142589] ? show_regs.cold.9+0x22/0x2f [ 5204.144576] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5204.147958] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5204.149923] ? watchdog+0x40/0x40 [ 5204.150975] ? hrtimer_interrupt+0x128/0x2c0 [ 5204.152561] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5204.155016] ? apic_timer_interrupt+0xf/0x20 [ 5204.158338] [ 5204.159816] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5204.165205] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5204.167127] do_raw_spin_lock+0x177/0x190 [ 5204.168191] _raw_spin_lock+0x21/0x30 [ 5204.169183] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5204.179557] process_one_work+0x2c8/0x700 [ 5204.181044] worker_thread+0x296/0x6e0 [ 5204.183509] ? rescuer_thread+0x570/0x570 [ 5204.185425] kthread+0x1d1/0x200 [ 5204.186420] ? set_kthread_struct+0x70/0x70 [ 5204.188587] ret_from_fork+0x1f/0x30 [ 5212.059097] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 5212.068369] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5212.103411] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5212.113878] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5212.129957] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5212.141779] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5212.155468] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5212.160056] RAX: 0000000000005b44 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5212.168359] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5212.180677] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5212.183220] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5212.187003] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5212.189709] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5212.193001] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5212.200491] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5212.211011] Call Trace: [ 5212.215003] [ 5212.217697] ? show_regs.cold.9+0x22/0x2f [ 5212.219056] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5212.230437] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5212.231961] ? watchdog+0x40/0x40 [ 5212.249426] ? hrtimer_interrupt+0x128/0x2c0 [ 5212.256470] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5212.264319] ? apic_timer_interrupt+0xf/0x20 [ 5212.267427] [ 5212.268735] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5212.272384] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5212.274713] do_raw_spin_lock+0x177/0x190 [ 5212.278068] _raw_spin_lock+0x21/0x30 [ 5212.282689] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5212.287298] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5212.293004] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5212.298972] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5212.304217] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5212.314157] ? woken_wake_function+0x30/0x30 [ 5212.315893] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5212.317537] kthread+0x1d1/0x200 [ 5212.318320] ? set_kthread_struct+0x70/0x70 [ 5212.330018] ret_from_fork+0x1f/0x30 [ 5220.056475] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 5220.060665] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5220.082131] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5220.085962] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5220.088050] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5220.089963] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5220.097824] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5220.101004] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5220.107114] RDX: 0000000000007d23 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5220.110370] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5220.116584] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5220.121339] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5220.124901] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5220.127666] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5220.133210] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5220.141417] Call Trace: [ 5220.146910] [ 5220.147460] ? show_regs.cold.9+0x22/0x2f [ 5220.149997] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5220.152146] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5220.155229] ? watchdog+0x40/0x40 [ 5220.156680] ? hrtimer_interrupt+0x128/0x2c0 [ 5220.159700] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5220.160980] ? apic_timer_interrupt+0xf/0x20 [ 5220.162871] [ 5220.163759] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5220.165459] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5220.167805] do_raw_spin_lock+0x177/0x190 [ 5220.169706] _raw_spin_lock+0x21/0x30 [ 5220.171457] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5220.178385] obd_mod_free+0x46/0x140 [ptlrpc] [ 5220.181900] mdc_commit_open+0x91/0xb0 [mdc] [ 5220.184053] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5220.186427] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5220.189088] after_reply+0xe2c/0x1780 [ptlrpc] [ 5220.194325] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5220.197237] ? __next_timer_interrupt+0x160/0x160 [ 5220.200774] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5220.206098] ? do_wait_intr+0xf0/0xf0 [ 5220.208467] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5220.211156] kthread+0x1d1/0x200 [ 5220.213427] ? set_kthread_struct+0x70/0x70 [ 5220.215286] ret_from_fork+0x1f/0x30 [ 5223.396917] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2710s! [ 5223.414362] Showing busy workqueues and worker pools: [ 5223.421419] workqueue events: flags=0x0 [ 5223.428212] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5223.433355] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5223.443313] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5223.456864] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5223.466146] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5223.473891] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5223.487993] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5223.495771] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5223.504273] workqueue events_long: flags=0x0 [ 5223.511230] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5223.520921] pending: lu_objects_flush [obdclass] [ 5223.528276] workqueue events_unbound: flags=0x2 [ 5223.537599] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5223.544394] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5223.550241] workqueue events_power_efficient: flags=0x80 [ 5223.557130] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5223.566966] pending: neigh_periodic_work, do_cache_clean [ 5223.577931] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5223.587759] pending: neigh_periodic_work, check_lifetime [ 5223.595145] workqueue mm_percpu_wq: flags=0x8 [ 5223.599411] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5223.607072] pending: vmstat_update, lru_add_drain_per_cpu [ 5223.612473] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5223.622555] pending: vmstat_update [ 5223.628638] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5223.643458] pending: lru_add_drain_per_cpu BAR(48) [ 5223.651110] workqueue ipv6_addrconf: flags=0x40008 [ 5223.659515] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5223.667455] pending: addrconf_verify_work [ 5223.673316] workqueue knbd0-recv: flags=0x1a [ 5223.677360] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5223.682766] in-flight: 650:recv_work [ 5223.684166] workqueue ptlrpc_pinger: flags=0x2 [ 5223.690136] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5223.698676] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5223.705530] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2710s workers=4 idle: 36 466 29757 [ 5223.715452] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5213s workers=2 idle: 652 [ 5228.062042] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 5228.065145] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5228.084428] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5228.089880] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5228.093559] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5228.095586] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5228.104044] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5228.107077] RAX: 0000000000000b0e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5228.110021] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5228.112977] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5228.115639] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5228.118192] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5228.121849] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5228.125594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5228.128123] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5228.131354] Call Trace: [ 5228.132617] [ 5228.133532] ? show_regs.cold.9+0x22/0x2f [ 5228.135313] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5228.137798] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5228.140038] ? watchdog+0x40/0x40 [ 5228.141545] ? hrtimer_interrupt+0x128/0x2c0 [ 5228.143647] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5228.145816] ? apic_timer_interrupt+0xf/0x20 [ 5228.147887] [ 5228.148871] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5228.151419] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5228.154731] do_raw_spin_lock+0x177/0x190 [ 5228.157071] _raw_spin_lock+0x21/0x30 [ 5228.158841] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5228.161028] ? do_raw_spin_unlock+0x75/0x190 [ 5228.162892] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5228.165040] ? do_wait_intr+0xf0/0xf0 [ 5228.166845] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5228.169617] mdc_reint+0x2d/0x110 [mdc] [ 5228.171733] mdc_unlink+0x658/0x870 [mdc] [ 5228.173675] lmv_unlink+0x34d/0xbd0 [lmv] [ 5228.175690] ll_unlink+0x24d/0x7f0 [lustre] [ 5228.177776] vfs_unlink+0x202/0x330 [ 5228.179350] do_unlinkat+0x2fa/0x460 [ 5228.181124] __x64_sys_unlink+0x37/0x50 [ 5228.182983] do_syscall_64+0xc1/0x440 [ 5228.184734] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5228.186994] RIP: 0033:0x7f83e5f5c10b [ 5228.188919] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5228.197231] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5228.200505] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5228.203579] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5228.206894] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5228.210006] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5228.213898] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5232.068534] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5232.079754] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5232.109937] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5232.115513] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5232.119558] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5232.125939] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5232.129742] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5232.139451] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5232.142447] RAX: 0000000000006d82 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5232.146322] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5232.156770] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5232.161973] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5232.166239] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5232.172030] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5232.179111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5232.186065] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5232.192086] Call Trace: [ 5232.194802] [ 5232.195614] ? show_regs.cold.9+0x22/0x2f [ 5232.198937] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5232.201017] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5232.203151] ? watchdog+0x40/0x40 [ 5232.204615] ? hrtimer_interrupt+0x128/0x2c0 [ 5232.206537] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5232.213132] ? apic_timer_interrupt+0xf/0x20 [ 5232.216781] [ 5232.219612] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5232.222070] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5232.227388] do_raw_spin_lock+0x177/0x190 [ 5232.229119] _raw_spin_lock+0x21/0x30 [ 5232.231054] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5232.234448] process_one_work+0x2c8/0x700 [ 5232.236522] worker_thread+0x296/0x6e0 [ 5232.238515] ? rescuer_thread+0x570/0x570 [ 5232.240513] kthread+0x1d1/0x200 [ 5232.244204] ? set_kthread_struct+0x70/0x70 [ 5232.245693] ret_from_fork+0x1f/0x30 [ 5240.059065] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 5240.062842] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5240.079140] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5240.084888] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5240.088381] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5240.089885] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5240.104387] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5240.110110] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5240.114264] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5240.122757] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5240.130015] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5240.134426] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5240.143655] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5240.150595] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5240.160752] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5240.168577] Call Trace: [ 5240.169713] [ 5240.172090] ? show_regs.cold.9+0x22/0x2f [ 5240.177079] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5240.178751] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5240.184082] ? watchdog+0x40/0x40 [ 5240.187055] ? hrtimer_interrupt+0x128/0x2c0 [ 5240.189039] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5240.193387] ? apic_timer_interrupt+0xf/0x20 [ 5240.200138] [ 5240.202875] ? native_safe_halt+0x2e/0x50 [ 5240.206239] kvm_wait+0xcd/0xe0 [ 5240.209719] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5240.218501] do_raw_spin_lock+0x177/0x190 [ 5240.223301] _raw_spin_lock+0x21/0x30 [ 5240.227514] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5240.233098] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5240.236633] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5240.246886] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5240.252745] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5240.259098] ? woken_wake_function+0x30/0x30 [ 5240.263303] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5240.269669] kthread+0x1d1/0x200 [ 5240.275669] ? set_kthread_struct+0x70/0x70 [ 5240.283192] ret_from_fork+0x1f/0x30 [ 5248.057437] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 5248.064325] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5248.109550] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5248.119848] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5248.125133] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5248.128558] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5248.142412] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5248.149410] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5248.156372] RDX: 00000000000016fc RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5248.160489] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5248.172184] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5248.180072] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5248.185132] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5248.190896] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5248.194295] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5248.203205] Call Trace: [ 5248.206289] [ 5248.209838] ? show_regs.cold.9+0x22/0x2f [ 5248.217349] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5248.221809] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5248.227076] ? watchdog+0x40/0x40 [ 5248.231248] ? hrtimer_interrupt+0x128/0x2c0 [ 5248.234186] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5248.241954] ? apic_timer_interrupt+0xf/0x20 [ 5248.243410] [ 5248.247202] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5248.249217] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5248.259630] do_raw_spin_lock+0x177/0x190 [ 5248.262899] _raw_spin_lock+0x21/0x30 [ 5248.266759] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5248.268696] obd_mod_free+0x46/0x140 [ptlrpc] [ 5248.276851] mdc_commit_open+0x91/0xb0 [mdc] [ 5248.281180] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5248.284237] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5248.287997] after_reply+0xe2c/0x1780 [ptlrpc] [ 5248.293017] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5248.297156] ? __next_timer_interrupt+0x160/0x160 [ 5248.299969] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5248.302006] ? do_wait_intr+0xf0/0xf0 [ 5248.303554] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5248.306267] kthread+0x1d1/0x200 [ 5248.307938] ? set_kthread_struct+0x70/0x70 [ 5248.310344] ret_from_fork+0x1f/0x30 [ 5254.113049] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2740s! [ 5254.115904] Showing busy workqueues and worker pools: [ 5254.117959] workqueue events: flags=0x0 [ 5254.119190] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5254.121493] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5254.125431] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5254.127886] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5254.130246] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5254.132788] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5254.135750] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5254.138208] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5254.141680] workqueue events_long: flags=0x0 [ 5254.143411] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5254.145793] pending: lu_objects_flush [obdclass] [ 5254.148009] workqueue events_unbound: flags=0x2 [ 5254.149906] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5254.152358] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5254.154209] workqueue events_power_efficient: flags=0x80 [ 5254.155859] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5254.158228] pending: neigh_periodic_work, do_cache_clean [ 5254.160225] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5254.162689] pending: neigh_periodic_work, check_lifetime [ 5254.164761] workqueue mm_percpu_wq: flags=0x8 [ 5254.166265] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5254.168278] pending: vmstat_update, lru_add_drain_per_cpu [ 5254.170113] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5254.172207] pending: vmstat_update [ 5254.173407] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5254.175877] pending: lru_add_drain_per_cpu BAR(48) [ 5254.177669] workqueue ipv6_addrconf: flags=0x40008 [ 5254.179340] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5254.181649] pending: addrconf_verify_work [ 5254.183107] workqueue knbd0-recv: flags=0x1a [ 5254.184467] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5254.186546] in-flight: 650:recv_work [ 5254.188012] workqueue ptlrpc_pinger: flags=0x2 [ 5254.189959] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5254.191925] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5254.195916] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2741s workers=4 idle: 36 466 29757 [ 5254.199473] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5244s workers=2 idle: 652 [ 5256.062025] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 5256.065720] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5256.085808] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5256.089577] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5256.093382] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5256.095025] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5256.103038] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5256.106342] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5256.109388] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5256.112745] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5256.116766] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5256.120171] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5256.122354] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5256.127430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5256.131304] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5256.134614] Call Trace: [ 5256.137437] [ 5256.137911] ? show_regs.cold.9+0x22/0x2f [ 5256.139058] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5256.140603] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5256.142137] ? watchdog+0x40/0x40 [ 5256.143103] ? hrtimer_interrupt+0x128/0x2c0 [ 5256.144166] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5256.145610] ? apic_timer_interrupt+0xf/0x20 [ 5256.146722] [ 5256.147325] ? native_safe_halt+0x2e/0x50 [ 5256.148285] kvm_wait+0xcd/0xe0 [ 5256.149097] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5256.150462] do_raw_spin_lock+0x177/0x190 [ 5256.151495] _raw_spin_lock+0x21/0x30 [ 5256.152447] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5256.154253] ? do_raw_spin_unlock+0x75/0x190 [ 5256.155823] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5256.158469] ? do_wait_intr+0xf0/0xf0 [ 5256.160398] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5256.163416] mdc_reint+0x2d/0x110 [mdc] [ 5256.165360] mdc_unlink+0x658/0x870 [mdc] [ 5256.167392] lmv_unlink+0x34d/0xbd0 [lmv] [ 5256.169067] ll_unlink+0x24d/0x7f0 [lustre] [ 5256.170774] vfs_unlink+0x202/0x330 [ 5256.172573] do_unlinkat+0x2fa/0x460 [ 5256.174194] __x64_sys_unlink+0x37/0x50 [ 5256.175912] do_syscall_64+0xc1/0x440 [ 5256.177438] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5256.179489] RIP: 0033:0x7f83e5f5c10b [ 5256.181263] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5256.187532] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5256.189606] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5256.192007] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5256.195498] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5256.198912] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5256.202484] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5260.060961] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5260.075977] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5260.117072] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5260.124823] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5260.137862] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5260.147404] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5260.150580] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5260.165974] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5260.172808] RAX: 0000000000007a10 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5260.184174] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5260.189650] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5260.198021] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5260.203657] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5260.211873] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5260.223547] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5260.241283] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5260.244732] Call Trace: [ 5260.245878] [ 5260.246871] ? show_regs.cold.9+0x22/0x2f [ 5260.248709] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5260.251341] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5260.253469] ? watchdog+0x40/0x40 [ 5260.255860] ? hrtimer_interrupt+0x128/0x2c0 [ 5260.257885] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5260.267653] ? apic_timer_interrupt+0xf/0x20 [ 5260.272779] [ 5260.274764] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5260.277849] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5260.281986] do_raw_spin_lock+0x177/0x190 [ 5260.284231] _raw_spin_lock+0x21/0x30 [ 5260.285573] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5260.293485] process_one_work+0x2c8/0x700 [ 5260.297947] worker_thread+0x296/0x6e0 [ 5260.308917] ? rescuer_thread+0x570/0x570 [ 5260.312668] kthread+0x1d1/0x200 [ 5260.313800] ? set_kthread_struct+0x70/0x70 [ 5260.316382] ret_from_fork+0x1f/0x30 [ 5268.059024] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 5268.068895] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5268.096714] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5268.106668] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5268.114553] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5268.120318] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5268.135590] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5268.139294] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5268.143471] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5268.149472] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5268.156327] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5268.159844] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5268.163427] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5268.168662] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5268.171484] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5268.176771] Call Trace: [ 5268.178873] [ 5268.179897] ? show_regs.cold.9+0x22/0x2f [ 5268.181294] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5268.184464] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5268.187331] ? watchdog+0x40/0x40 [ 5268.188674] ? hrtimer_interrupt+0x128/0x2c0 [ 5268.190410] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5268.194921] ? apic_timer_interrupt+0xf/0x20 [ 5268.198095] [ 5268.198996] ? native_safe_halt+0x2e/0x50 [ 5268.201382] kvm_wait+0xcd/0xe0 [ 5268.202732] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5268.204550] do_raw_spin_lock+0x177/0x190 [ 5268.206214] _raw_spin_lock+0x21/0x30 [ 5268.207655] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5268.210213] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5268.212642] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5268.215314] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5268.218291] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5268.223825] ? woken_wake_function+0x30/0x30 [ 5268.229653] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5268.231961] kthread+0x1d1/0x200 [ 5268.232838] ? set_kthread_struct+0x70/0x70 [ 5268.235262] ret_from_fork+0x1f/0x30 [ 5280.056465] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 5280.062056] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5280.090547] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5280.095203] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5280.099261] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5280.101433] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5280.108793] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5280.111925] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5280.114543] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5280.117722] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5280.120872] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5280.124599] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5280.129181] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5280.132900] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5280.135368] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5280.140196] Call Trace: [ 5280.141173] [ 5280.141877] ? show_regs.cold.9+0x22/0x2f [ 5280.143686] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5280.145419] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5280.146871] ? watchdog+0x40/0x40 [ 5280.148228] ? hrtimer_interrupt+0x128/0x2c0 [ 5280.150112] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5280.151835] ? apic_timer_interrupt+0xf/0x20 [ 5280.154518] [ 5280.155549] ? native_safe_halt+0x2e/0x50 [ 5280.157103] kvm_wait+0xcd/0xe0 [ 5280.158223] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5280.160138] do_raw_spin_lock+0x177/0x190 [ 5280.161718] _raw_spin_lock+0x21/0x30 [ 5280.163184] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5280.165286] obd_mod_free+0x46/0x140 [ptlrpc] [ 5280.167190] mdc_commit_open+0x91/0xb0 [mdc] [ 5280.168856] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5280.171065] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5280.173399] after_reply+0xe2c/0x1780 [ptlrpc] [ 5280.175390] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5280.177567] ? __next_timer_interrupt+0x160/0x160 [ 5280.179320] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5280.181077] ? do_wait_intr+0xf0/0xf0 [ 5280.182475] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5280.184673] kthread+0x1d1/0x200 [ 5280.185851] ? set_kthread_struct+0x70/0x70 [ 5280.188082] ret_from_fork+0x1f/0x30 [ 5281.033009] rcu: INFO: rcu_sched self-detected stall on CPU [ 5281.043682] rcu: 0-....: (2075764 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 5281.058685] (t=3300099 jiffies g=262441 q=58086) [ 5281.068784] rcu: rcu_sched kthread starved for 2769296 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 5281.083200] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5281.089819] rcu: RCU grace-period kthread stack dump: [ 5281.098828] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5281.107678] Call Trace: [ 5281.109440] __schedule+0x351/0xcb0 [ 5281.111267] schedule+0xc0/0x180 [ 5281.112556] schedule_timeout+0xb4/0x190 [ 5281.115813] ? do_raw_spin_unlock+0x75/0x190 [ 5281.126266] ? __next_timer_interrupt+0x160/0x160 [ 5281.133123] ? prepare_to_swait_event+0xf3/0x140 [ 5281.136217] rcu_gp_kthread+0x8dd/0x15b0 [ 5281.140685] ? force_qs_rnp+0x2c0/0x2c0 [ 5281.142807] kthread+0x1d1/0x200 [ 5281.146392] ? set_kthread_struct+0x70/0x70 [ 5281.149857] ret_from_fork+0x1f/0x30 [ 5281.151192] rcu: Stack dump where RCU GP kthread last ran: [ 5281.155915] NMI backtrace for cpu 0 [ 5281.159046] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5281.167708] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5281.174824] Call Trace: [ 5281.176853] [ 5281.177705] ? dump_stack+0xbb/0x10e [ 5281.179145] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5281.181468] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5281.183539] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5281.185806] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5281.187894] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5281.189614] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 5281.191352] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 5281.193408] ? update_process_times+0x6d/0xc0 [ 5281.195852] ? tick_sched_handle.isra.18+0x27/0x90 [ 5281.197661] ? tick_sched_timer+0x53/0xb0 [ 5281.199873] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5281.202543] ? tick_sched_handle.isra.18+0x90/0x90 [ 5281.205494] ? hrtimer_interrupt+0x128/0x2c0 [ 5281.207097] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5281.208976] ? apic_timer_interrupt+0xf/0x20 [ 5281.211454] [ 5281.212366] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5281.215587] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5281.218997] ? do_raw_spin_lock+0x177/0x190 [ 5281.221604] ? _raw_spin_lock+0x21/0x30 [ 5281.223431] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5281.228406] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5281.233040] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5281.235915] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5281.240390] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5281.244846] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5281.256692] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5281.265299] ? __next_timer_interrupt+0x160/0x160 [ 5281.271686] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5281.279631] ? do_wait_intr+0xf0/0xf0 [ 5281.285494] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5281.293078] ? kthread+0x1d1/0x200 [ 5281.296279] ? set_kthread_struct+0x70/0x70 [ 5281.301375] ? ret_from_fork+0x1f/0x30 [ 5281.305287] NMI backtrace for cpu 0 [ 5281.310669] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5281.317123] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5281.327370] Call Trace: [ 5281.329844] [ 5281.332306] ? dump_stack+0xbb/0x10e [ 5281.335450] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5281.340085] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5281.345630] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5281.351059] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5281.357072] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5281.362311] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5281.366773] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5281.376261] ? update_process_times+0x6d/0xc0 [ 5281.379601] ? tick_sched_handle.isra.18+0x27/0x90 [ 5281.384921] ? tick_sched_timer+0x53/0xb0 [ 5281.388852] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5281.396600] ? tick_sched_handle.isra.18+0x90/0x90 [ 5281.404491] ? hrtimer_interrupt+0x128/0x2c0 [ 5281.409063] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5281.413463] ? apic_timer_interrupt+0xf/0x20 [ 5281.414688] [ 5281.415347] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5281.426184] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5281.430233] ? do_raw_spin_lock+0x177/0x190 [ 5281.433453] ? _raw_spin_lock+0x21/0x30 [ 5281.436736] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5281.440066] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5281.446519] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5281.450437] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5281.454310] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5281.461665] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5281.467121] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5281.471486] ? __next_timer_interrupt+0x160/0x160 [ 5281.474586] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5281.476697] ? do_wait_intr+0xf0/0xf0 [ 5281.478242] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5281.480868] ? kthread+0x1d1/0x200 [ 5281.482475] ? set_kthread_struct+0x70/0x70 [ 5281.484243] ? ret_from_fork+0x1f/0x30 [ 5284.062010] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 5284.067022] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5284.103824] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5284.109267] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5284.113669] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5284.119137] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5284.130866] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5284.139687] RAX: 00000000000061fd RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5284.143192] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5284.146016] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5284.148843] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5284.151601] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5284.154257] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5284.156581] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5284.158595] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5284.161098] Call Trace: [ 5284.161872] [ 5284.162600] ? show_regs.cold.9+0x22/0x2f [ 5284.163871] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5284.165498] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5284.167050] ? watchdog+0x40/0x40 [ 5284.168350] ? hrtimer_interrupt+0x128/0x2c0 [ 5284.169741] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5284.171068] ? apic_timer_interrupt+0xf/0x20 [ 5284.172268] [ 5284.172820] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5284.181429] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5284.183671] do_raw_spin_lock+0x177/0x190 [ 5284.185521] _raw_spin_lock+0x21/0x30 [ 5284.187571] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5284.190031] ? do_raw_spin_unlock+0x75/0x190 [ 5284.192686] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5284.195761] ? do_wait_intr+0xf0/0xf0 [ 5284.197878] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5284.202040] mdc_reint+0x2d/0x110 [mdc] [ 5284.206268] mdc_unlink+0x658/0x870 [mdc] [ 5284.211006] lmv_unlink+0x34d/0xbd0 [lmv] [ 5284.216843] ll_unlink+0x24d/0x7f0 [lustre] [ 5284.219220] vfs_unlink+0x202/0x330 [ 5284.225329] do_unlinkat+0x2fa/0x460 [ 5284.226446] __x64_sys_unlink+0x37/0x50 [ 5284.228066] do_syscall_64+0xc1/0x440 [ 5284.230473] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5284.236567] RIP: 0033:0x7f83e5f5c10b [ 5284.237908] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5284.256000] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5284.260387] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5284.264756] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5284.269751] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5284.271409] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5284.275814] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5284.321034] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2771s! [ 5284.324198] Showing busy workqueues and worker pools: [ 5284.326249] workqueue events: flags=0x0 [ 5284.327843] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5284.331015] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5284.335775] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5284.339025] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5284.342582] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5284.345363] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5284.349385] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5284.352360] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5284.357107] workqueue events_long: flags=0x0 [ 5284.358784] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5284.361971] pending: lu_objects_flush [obdclass] [ 5284.365102] workqueue events_unbound: flags=0x2 [ 5284.367418] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5284.369672] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5284.372946] workqueue events_power_efficient: flags=0x80 [ 5284.374984] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5284.377411] pending: neigh_periodic_work, do_cache_clean [ 5284.379974] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5284.382771] pending: neigh_periodic_work, check_lifetime [ 5284.385244] workqueue mm_percpu_wq: flags=0x8 [ 5284.386418] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5284.388643] pending: vmstat_update, lru_add_drain_per_cpu [ 5284.390825] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5284.393303] pending: vmstat_update [ 5284.395116] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5284.397625] pending: lru_add_drain_per_cpu BAR(48) [ 5284.399229] workqueue ipv6_addrconf: flags=0x40008 [ 5284.400987] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5284.403687] pending: addrconf_verify_work [ 5284.405016] workqueue knbd0-recv: flags=0x1a [ 5284.406781] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5284.409567] in-flight: 650:recv_work [ 5284.410986] workqueue ptlrpc_pinger: flags=0x2 [ 5284.412222] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5284.414207] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5284.416415] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2771s workers=4 idle: 36 466 29757 [ 5284.419839] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5274s workers=2 idle: 652 [ 5288.060044] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5288.063988] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5288.102059] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5288.110299] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5288.119270] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5288.126349] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5288.129594] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5288.149512] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5288.157198] RAX: 000000000000762c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5288.162247] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5288.166326] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5288.170322] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5288.173409] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5288.178832] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5288.185189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5288.192070] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5288.197858] Call Trace: [ 5288.199597] [ 5288.201118] ? show_regs.cold.9+0x22/0x2f [ 5288.203183] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5288.208969] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5288.212330] ? watchdog+0x40/0x40 [ 5288.218342] ? hrtimer_interrupt+0x128/0x2c0 [ 5288.222066] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5288.227331] ? apic_timer_interrupt+0xf/0x20 [ 5288.230213] [ 5288.231557] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5288.236337] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5288.240360] do_raw_spin_lock+0x177/0x190 [ 5288.245574] _raw_spin_lock+0x21/0x30 [ 5288.248367] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5288.253283] process_one_work+0x2c8/0x700 [ 5288.256192] worker_thread+0x296/0x6e0 [ 5288.258646] ? rescuer_thread+0x570/0x570 [ 5288.261506] kthread+0x1d1/0x200 [ 5288.263281] ? set_kthread_struct+0x70/0x70 [ 5288.265219] ret_from_fork+0x1f/0x30 [ 5296.059013] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 5296.062477] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5296.091422] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5296.096300] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5296.103184] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5296.106714] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5296.119210] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5296.125472] RAX: 000000000000418b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5296.134728] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5296.141396] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5296.143985] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5296.157590] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5296.165598] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5296.172425] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5296.184459] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5296.192139] Call Trace: [ 5296.193055] [ 5296.193798] ? show_regs.cold.9+0x22/0x2f [ 5296.196706] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5296.202433] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5296.205642] ? watchdog+0x40/0x40 [ 5296.210471] ? hrtimer_interrupt+0x128/0x2c0 [ 5296.213075] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5296.217684] ? apic_timer_interrupt+0xf/0x20 [ 5296.221057] [ 5296.222942] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5296.227009] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5296.235109] do_raw_spin_lock+0x177/0x190 [ 5296.238038] _raw_spin_lock+0x21/0x30 [ 5296.241706] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5296.248109] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5296.250266] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5296.255253] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5296.263345] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5296.267057] ? woken_wake_function+0x30/0x30 [ 5296.272025] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5296.277552] kthread+0x1d1/0x200 [ 5296.282026] ? set_kthread_struct+0x70/0x70 [ 5296.287080] ret_from_fork+0x1f/0x30 [ 5312.056074] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 5312.061056] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5312.066549] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 5312.086225] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5312.094897] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5312.094976] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5312.094983] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5312.094986] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5312.095001] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5312.095005] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5312.095009] RAX: 0000000000003d93 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5312.095012] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5312.095014] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5312.095016] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5312.095019] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5312.095022] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5312.095025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5312.095028] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5312.095037] Call Trace: [ 5312.095042] [ 5312.095048] ? show_regs.cold.9+0x22/0x2f [ 5312.102180] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5312.102190] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5312.102206] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5312.102210] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5312.102216] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5312.102219] RDX: 0000000000003eed RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5312.102222] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5312.125901] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5312.125921] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5312.125925] ? watchdog+0x40/0x40 [ 5312.125929] ? hrtimer_interrupt+0x128/0x2c0 [ 5312.125933] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5312.125941] ? apic_timer_interrupt+0xf/0x20 [ 5312.125946] [ 5312.125950] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5312.125957] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5312.125962] do_raw_spin_lock+0x177/0x190 [ 5312.125969] _raw_spin_lock+0x21/0x30 [ 5312.125975] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5312.131728] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5312.131738] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5312.131742] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5312.131745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5312.131747] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5312.131755] Call Trace: [ 5312.131761] [ 5312.131767] ? show_regs.cold.9+0x22/0x2f [ 5312.131779] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5312.131785] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5312.131790] ? watchdog+0x40/0x40 [ 5312.131794] ? hrtimer_interrupt+0x128/0x2c0 [ 5312.131798] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5312.131808] ? apic_timer_interrupt+0xf/0x20 [ 5312.131812] [ 5312.131815] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5312.131821] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5312.131826] do_raw_spin_lock+0x177/0x190 [ 5312.131831] _raw_spin_lock+0x21/0x30 [ 5312.131837] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5312.132233] obd_mod_free+0x46/0x140 [ptlrpc] [ 5312.132611] mdc_commit_open+0x91/0xb0 [mdc] [ 5312.132662] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5312.132867] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5312.133087] after_reply+0xe2c/0x1780 [ptlrpc] [ 5312.133476] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5312.133862] ? __next_timer_interrupt+0x160/0x160 [ 5312.133874] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5312.134090] ? do_wait_intr+0xf0/0xf0 [ 5312.134099] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5312.134359] kthread+0x1d1/0x200 [ 5312.134365] ? set_kthread_struct+0x70/0x70 [ 5312.134369] ret_from_fork+0x1f/0x30 [ 5312.391083] ? do_raw_spin_unlock+0x75/0x190 [ 5312.395467] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5312.401064] ? do_wait_intr+0xf0/0xf0 [ 5312.403390] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5312.406119] mdc_reint+0x2d/0x110 [mdc] [ 5312.408030] mdc_unlink+0x658/0x870 [mdc] [ 5312.409949] lmv_unlink+0x34d/0xbd0 [lmv] [ 5312.411983] ll_unlink+0x24d/0x7f0 [lustre] [ 5312.413752] vfs_unlink+0x202/0x330 [ 5312.415363] do_unlinkat+0x2fa/0x460 [ 5312.422014] __x64_sys_unlink+0x37/0x50 [ 5312.424303] do_syscall_64+0xc1/0x440 [ 5312.430126] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5312.437350] RIP: 0033:0x7f83e5f5c10b [ 5312.443122] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5312.463199] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5312.470496] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5312.478729] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5312.486724] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5312.495937] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5312.500462] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5314.533237] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2801s! [ 5314.540961] Showing busy workqueues and worker pools: [ 5314.544073] workqueue events: flags=0x0 [ 5314.547538] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5314.551893] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5314.557799] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5314.561899] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5314.571675] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5314.577549] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5314.582515] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5314.585753] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5314.590558] workqueue events_long: flags=0x0 [ 5314.592049] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5314.594569] pending: lu_objects_flush [obdclass] [ 5314.596745] workqueue events_unbound: flags=0x2 [ 5314.598549] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5314.600755] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5314.602793] workqueue events_power_efficient: flags=0x80 [ 5314.605481] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5314.608190] pending: neigh_periodic_work, do_cache_clean [ 5314.612768] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5314.616997] pending: neigh_periodic_work, check_lifetime [ 5314.621123] workqueue mm_percpu_wq: flags=0x8 [ 5314.623790] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5314.628954] pending: vmstat_update, lru_add_drain_per_cpu [ 5314.632692] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5314.637935] pending: vmstat_update [ 5314.640715] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5314.644626] pending: lru_add_drain_per_cpu BAR(48) [ 5314.647992] workqueue ipv6_addrconf: flags=0x40008 [ 5314.651993] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5314.657017] pending: addrconf_verify_work [ 5314.662516] workqueue knbd0-recv: flags=0x1a [ 5314.664615] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5314.679776] in-flight: 650:recv_work [ 5314.691104] workqueue ptlrpc_pinger: flags=0x2 [ 5314.697257] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5314.704773] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5314.711168] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2801s workers=4 idle: 36 466 29757 [ 5314.718281] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5304s workers=2 idle: 652 [ 5316.060180] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5316.067305] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5316.096924] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5316.104109] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5316.109240] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5316.114929] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5316.117796] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5316.128935] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5316.133432] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5316.137676] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5316.142197] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5316.147273] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5316.152129] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5316.157174] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5316.163516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5316.168110] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5316.173057] Call Trace: [ 5316.175050] [ 5316.176696] ? show_regs.cold.9+0x22/0x2f [ 5316.179915] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5316.184562] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5316.188153] ? watchdog+0x40/0x40 [ 5316.189478] ? hrtimer_interrupt+0x128/0x2c0 [ 5316.190681] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5316.193631] ? apic_timer_interrupt+0xf/0x20 [ 5316.195403] [ 5316.197411] ? native_safe_halt+0x2e/0x50 [ 5316.200032] kvm_wait+0xcd/0xe0 [ 5316.202853] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5316.210186] do_raw_spin_lock+0x177/0x190 [ 5316.217078] _raw_spin_lock+0x21/0x30 [ 5316.223116] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5316.230448] process_one_work+0x2c8/0x700 [ 5316.236320] worker_thread+0x296/0x6e0 [ 5316.237589] ? rescuer_thread+0x570/0x570 [ 5316.241766] kthread+0x1d1/0x200 [ 5316.244737] ? set_kthread_struct+0x70/0x70 [ 5316.248121] ret_from_fork+0x1f/0x30 [ 5324.060117] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 5324.063534] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5324.095711] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5324.101201] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5324.106967] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5324.112513] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5324.130458] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5324.134182] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5324.137078] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5324.145222] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5324.154075] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5324.165000] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5324.175204] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5324.179963] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5324.183764] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5324.186072] Call Trace: [ 5324.189226] [ 5324.190137] ? show_regs.cold.9+0x22/0x2f [ 5324.193386] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5324.198584] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5324.200124] ? watchdog+0x40/0x40 [ 5324.202396] ? hrtimer_interrupt+0x128/0x2c0 [ 5324.204697] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5324.206848] ? apic_timer_interrupt+0xf/0x20 [ 5324.208794] [ 5324.209666] ? native_safe_halt+0x2e/0x50 [ 5324.211407] kvm_wait+0xcd/0xe0 [ 5324.212884] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5324.219214] do_raw_spin_lock+0x177/0x190 [ 5324.220467] _raw_spin_lock+0x21/0x30 [ 5324.232463] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5324.234772] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5324.241220] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5324.245185] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5324.251479] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5324.259938] ? woken_wake_function+0x30/0x30 [ 5324.265402] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5324.272451] kthread+0x1d1/0x200 [ 5324.276607] ? set_kthread_struct+0x70/0x70 [ 5324.283945] ret_from_fork+0x1f/0x30 [ 5340.056899] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 5340.061950] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5340.062371] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 5340.087575] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5340.087589] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5340.087592] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5340.087606] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5340.087610] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5340.087621] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5340.087624] RDX: 00000000000047f0 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5340.087626] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5340.087628] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5340.087631] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5340.087634] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5340.087636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5340.087639] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5340.087647] Call Trace: [ 5340.087653] [ 5340.087658] ? show_regs.cold.9+0x22/0x2f [ 5340.087668] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5340.087681] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5340.087686] ? watchdog+0x40/0x40 [ 5340.087690] ? hrtimer_interrupt+0x128/0x2c0 [ 5340.087695] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5340.087706] ? apic_timer_interrupt+0xf/0x20 [ 5340.087711] [ 5340.087714] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5340.087719] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5340.087724] do_raw_spin_lock+0x177/0x190 [ 5340.087729] _raw_spin_lock+0x21/0x30 [ 5340.087740] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5340.088493] obd_mod_free+0x46/0x140 [ptlrpc] [ 5340.088734] mdc_commit_open+0x91/0xb0 [mdc] [ 5340.088770] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5340.089116] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5340.089494] after_reply+0xe2c/0x1780 [ptlrpc] [ 5340.089686] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5340.089899] ? __next_timer_interrupt+0x160/0x160 [ 5340.089908] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5340.093730] Modules linked in: lustre(O) [ 5340.099799] ? do_wait_intr+0xf0/0xf0 [ 5340.099876] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5340.100312] kthread+0x1d1/0x200 [ 5340.100320] ? set_kthread_struct+0x70/0x70 [ 5340.100323] ret_from_fork+0x1f/0x30 [ 5340.249941] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5340.280755] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5340.292960] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5340.298274] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5340.301823] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5340.312520] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5340.316946] RAX: 0000000000000809 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5340.320804] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5340.324796] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5340.328722] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5340.332984] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5340.336844] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5340.342707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5340.348951] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5340.361694] Call Trace: [ 5340.365598] [ 5340.366419] ? show_regs.cold.9+0x22/0x2f [ 5340.369523] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5340.372320] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5340.376590] ? watchdog+0x40/0x40 [ 5340.380709] ? hrtimer_interrupt+0x128/0x2c0 [ 5340.385339] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5340.387725] ? apic_timer_interrupt+0xf/0x20 [ 5340.391954] [ 5340.393691] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5340.399993] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5340.402587] do_raw_spin_lock+0x177/0x190 [ 5340.408791] _raw_spin_lock+0x21/0x30 [ 5340.413282] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5340.418607] ? do_raw_spin_unlock+0x75/0x190 [ 5340.423809] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5340.429490] ? do_wait_intr+0xf0/0xf0 [ 5340.432642] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5340.437833] mdc_reint+0x2d/0x110 [mdc] [ 5340.440668] mdc_unlink+0x658/0x870 [mdc] [ 5340.444233] lmv_unlink+0x34d/0xbd0 [lmv] [ 5340.446682] ll_unlink+0x24d/0x7f0 [lustre] [ 5340.448745] vfs_unlink+0x202/0x330 [ 5340.451093] do_unlinkat+0x2fa/0x460 [ 5340.453094] __x64_sys_unlink+0x37/0x50 [ 5340.455592] do_syscall_64+0xc1/0x440 [ 5340.456778] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5340.460567] RIP: 0033:0x7f83e5f5c10b [ 5340.462648] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5340.468814] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5340.471758] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5340.474299] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5340.476546] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5340.478623] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5340.483313] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5344.060174] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 5344.064258] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5344.086721] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5344.091726] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5344.095942] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5344.099854] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5344.102611] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5344.111626] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5344.115470] RAX: 0000000000006184 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5344.119440] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5344.122249] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5344.124668] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5344.126761] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5344.129729] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5344.132870] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5344.135497] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5344.138805] Call Trace: [ 5344.140215] [ 5344.140921] ? show_regs.cold.9+0x22/0x2f [ 5344.142862] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5344.145388] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5344.147808] ? watchdog+0x40/0x40 [ 5344.149458] ? hrtimer_interrupt+0x128/0x2c0 [ 5344.152920] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5344.154643] ? apic_timer_interrupt+0xf/0x20 [ 5344.156701] [ 5344.157662] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5344.160555] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5344.163149] do_raw_spin_lock+0x177/0x190 [ 5344.165270] _raw_spin_lock+0x21/0x30 [ 5344.167201] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5344.169690] process_one_work+0x2c8/0x700 [ 5344.171756] worker_thread+0x296/0x6e0 [ 5344.173838] ? rescuer_thread+0x570/0x570 [ 5344.175694] kthread+0x1d1/0x200 [ 5344.177416] ? set_kthread_struct+0x70/0x70 [ 5344.179290] ret_from_fork+0x1f/0x30 [ 5344.747144] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2831s! [ 5344.756149] Showing busy workqueues and worker pools: [ 5344.758369] workqueue events: flags=0x0 [ 5344.759562] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5344.764054] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5344.772066] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5344.778118] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5344.784666] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5344.788717] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5344.793301] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5344.799875] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5344.803489] workqueue events_long: flags=0x0 [ 5344.804863] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5344.809817] pending: lu_objects_flush [obdclass] [ 5344.816258] workqueue events_unbound: flags=0x2 [ 5344.818198] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5344.827063] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5344.833052] workqueue events_power_efficient: flags=0x80 [ 5344.836140] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5344.843151] pending: neigh_periodic_work, do_cache_clean [ 5344.846803] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5344.852974] pending: neigh_periodic_work, check_lifetime [ 5344.858694] workqueue mm_percpu_wq: flags=0x8 [ 5344.862033] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5344.867755] pending: vmstat_update, lru_add_drain_per_cpu [ 5344.870715] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5344.874262] pending: vmstat_update [ 5344.878025] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5344.884442] pending: lru_add_drain_per_cpu BAR(48) [ 5344.889398] workqueue ipv6_addrconf: flags=0x40008 [ 5344.891495] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5344.898440] pending: addrconf_verify_work [ 5344.901350] workqueue knbd0-recv: flags=0x1a [ 5344.902457] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5344.905079] in-flight: 650:recv_work [ 5344.906186] workqueue ptlrpc_pinger: flags=0x2 [ 5344.907466] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5344.910179] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5344.913046] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2831s workers=4 idle: 36 466 29757 [ 5344.916670] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5334s workers=2 idle: 652 [ 5352.063938] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 5352.067915] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5352.092051] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5352.098694] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5352.103730] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5352.107454] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5352.119773] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5352.124809] RAX: 0000000000004005 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5352.134777] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5352.141637] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5352.148114] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5352.157885] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5352.164163] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5352.168686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5352.174521] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5352.179111] Call Trace: [ 5352.180671] [ 5352.181969] ? show_regs.cold.9+0x22/0x2f [ 5352.183361] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5352.185118] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5352.189506] ? watchdog+0x40/0x40 [ 5352.190443] ? hrtimer_interrupt+0x128/0x2c0 [ 5352.192113] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5352.193796] ? apic_timer_interrupt+0xf/0x20 [ 5352.197494] [ 5352.199448] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5352.204973] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5352.208015] do_raw_spin_lock+0x177/0x190 [ 5352.211677] _raw_spin_lock+0x21/0x30 [ 5352.213320] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5352.219668] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5352.224937] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5352.231384] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5352.237540] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5352.240530] ? woken_wake_function+0x30/0x30 [ 5352.242540] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5352.249737] kthread+0x1d1/0x200 [ 5352.251345] ? set_kthread_struct+0x70/0x70 [ 5352.259168] ret_from_fork+0x1f/0x30 [ 5368.062055] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5368.072645] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5368.093703] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5368.099381] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5368.104054] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5368.107006] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5368.115465] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5368.118742] RAX: 000000000000260c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5368.121703] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5368.124592] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5368.127109] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5368.130563] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5368.134447] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5368.142780] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5368.149694] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5368.160479] Call Trace: [ 5368.163591] [ 5368.167764] ? show_regs.cold.9+0x22/0x2f [ 5368.172340] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5368.176843] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5368.191253] ? watchdog+0x40/0x40 [ 5368.195977] ? hrtimer_interrupt+0x128/0x2c0 [ 5368.205461] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5368.214654] ? apic_timer_interrupt+0xf/0x20 [ 5368.216926] [ 5368.223868] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5368.228532] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5368.236933] do_raw_spin_lock+0x177/0x190 [ 5368.247626] _raw_spin_lock+0x21/0x30 [ 5368.249166] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5368.256577] ? do_raw_spin_unlock+0x75/0x190 [ 5368.258741] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5368.261402] ? do_wait_intr+0xf0/0xf0 [ 5368.263285] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5368.267335] mdc_reint+0x2d/0x110 [mdc] [ 5368.271038] mdc_unlink+0x658/0x870 [mdc] [ 5368.273773] lmv_unlink+0x34d/0xbd0 [lmv] [ 5368.277378] ll_unlink+0x24d/0x7f0 [lustre] [ 5368.280138] vfs_unlink+0x202/0x330 [ 5368.281706] do_unlinkat+0x2fa/0x460 [ 5368.283509] __x64_sys_unlink+0x37/0x50 [ 5368.285543] do_syscall_64+0xc1/0x440 [ 5368.287955] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5368.290212] RIP: 0033:0x7f83e5f5c10b [ 5368.295929] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5368.321863] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5368.332713] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5368.340277] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5368.347813] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5368.352416] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5368.356297] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5372.056317] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 5372.062432] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 5372.075527] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5372.075605] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5372.075612] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5372.075615] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5372.075629] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5372.075634] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5372.075639] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5372.075641] RDX: 0000000000007afe RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5372.075644] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5372.075646] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5372.075648] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5372.075651] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5372.075653] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5372.075656] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5372.075663] Call Trace: [ 5372.075669] [ 5372.075675] ? show_regs.cold.9+0x22/0x2f [ 5372.075684] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5372.075689] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5372.075694] ? watchdog+0x40/0x40 [ 5372.075698] ? hrtimer_interrupt+0x128/0x2c0 [ 5372.076244] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5372.076257] ? apic_timer_interrupt+0xf/0x20 [ 5372.076261] [ 5372.076265] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5372.076270] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5372.076274] do_raw_spin_lock+0x177/0x190 [ 5372.076280] _raw_spin_lock+0x21/0x30 [ 5372.076287] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5372.076575] obd_mod_free+0x46/0x140 [ptlrpc] [ 5372.076793] mdc_commit_open+0x91/0xb0 [mdc] [ 5372.077048] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5372.077276] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5372.077485] after_reply+0xe2c/0x1780 [ptlrpc] [ 5372.077682] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5372.078368] ? __next_timer_interrupt+0x160/0x160 [ 5372.078379] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5372.079239] ? do_wait_intr+0xf0/0xf0 [ 5372.079250] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5372.079458] kthread+0x1d1/0x200 [ 5372.079464] ? set_kthread_struct+0x70/0x70 [ 5372.079468] ret_from_fork+0x1f/0x30 [ 5372.190018] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5372.209120] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5372.213741] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5372.217418] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5372.220281] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5372.222600] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5372.231205] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5372.235798] RAX: 0000000000005be6 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5372.241717] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5372.244693] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5372.247720] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5372.250290] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5372.252560] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5372.254956] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5372.256825] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5372.259568] Call Trace: [ 5372.260603] [ 5372.261664] ? show_regs.cold.9+0x22/0x2f [ 5372.263381] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5372.265248] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5372.266995] ? watchdog+0x40/0x40 [ 5372.268422] ? hrtimer_interrupt+0x128/0x2c0 [ 5372.270485] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5372.272632] ? apic_timer_interrupt+0xf/0x20 [ 5372.274655] [ 5372.275558] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5372.277704] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5372.282270] do_raw_spin_lock+0x177/0x190 [ 5372.283447] _raw_spin_lock+0x21/0x30 [ 5372.288697] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5372.290621] process_one_work+0x2c8/0x700 [ 5372.300764] worker_thread+0x296/0x6e0 [ 5372.302172] ? rescuer_thread+0x570/0x570 [ 5372.303687] kthread+0x1d1/0x200 [ 5372.305133] ? set_kthread_struct+0x70/0x70 [ 5372.306615] ret_from_fork+0x1f/0x30 [ 5374.951873] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2861s! [ 5374.956544] Showing busy workqueues and worker pools: [ 5374.961234] workqueue events: flags=0x0 [ 5374.964165] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5374.971891] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5374.980589] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5374.990241] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5375.004876] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5375.012643] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5375.022485] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5375.033154] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5375.039269] workqueue events_long: flags=0x0 [ 5375.046027] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5375.054432] pending: lu_objects_flush [obdclass] [ 5375.060923] workqueue events_unbound: flags=0x2 [ 5375.063665] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5375.071122] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5375.076622] workqueue events_power_efficient: flags=0x80 [ 5375.078339] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5375.090086] pending: neigh_periodic_work, do_cache_clean [ 5375.102676] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5375.111357] pending: neigh_periodic_work, check_lifetime [ 5375.126100] workqueue mm_percpu_wq: flags=0x8 [ 5375.129518] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5375.133533] pending: vmstat_update, lru_add_drain_per_cpu [ 5375.141286] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5375.147873] pending: vmstat_update [ 5375.149356] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5375.155259] pending: lru_add_drain_per_cpu BAR(48) [ 5375.160457] workqueue ipv6_addrconf: flags=0x40008 [ 5375.165700] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5375.172546] pending: addrconf_verify_work [ 5375.176964] workqueue knbd0-recv: flags=0x1a [ 5375.182471] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5375.189496] in-flight: 650:recv_work [ 5375.194563] workqueue ptlrpc_pinger: flags=0x2 [ 5375.196378] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5375.203238] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5375.212485] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2862s workers=4 idle: 36 466 29757 [ 5375.222611] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5365s workers=2 idle: 652 [ 5380.059279] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 5380.062630] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5380.082997] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5380.087623] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5380.091930] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5380.093889] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5380.101608] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5380.104005] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5380.107073] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5380.110089] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5380.113531] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5380.116954] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5380.120016] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5380.123403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5380.125988] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5380.129268] Call Trace: [ 5380.130296] [ 5380.131145] ? show_regs.cold.9+0x22/0x2f [ 5380.133155] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5380.135391] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5380.137180] ? watchdog+0x40/0x40 [ 5380.138353] ? hrtimer_interrupt+0x128/0x2c0 [ 5380.140422] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5380.143042] ? apic_timer_interrupt+0xf/0x20 [ 5380.145078] [ 5380.146308] ? native_safe_halt+0x2e/0x50 [ 5380.148345] kvm_wait+0xcd/0xe0 [ 5380.150426] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5380.153088] do_raw_spin_lock+0x177/0x190 [ 5380.155317] _raw_spin_lock+0x21/0x30 [ 5380.157322] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5380.160598] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5380.164110] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5380.166934] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5380.169612] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5380.172346] ? woken_wake_function+0x30/0x30 [ 5380.174323] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5380.176675] kthread+0x1d1/0x200 [ 5380.177850] ? set_kthread_struct+0x70/0x70 [ 5380.179367] ret_from_fork+0x1f/0x30 [ 5396.064661] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5396.070035] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5396.128439] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5396.139139] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5396.145078] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5396.148086] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5396.156639] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5396.160073] RAX: 000000000000711b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5396.163107] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5396.166507] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5396.169733] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5396.172752] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5396.175467] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5396.178493] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5396.180487] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5396.182968] Call Trace: [ 5396.184033] [ 5396.184800] ? show_regs.cold.9+0x22/0x2f [ 5396.186433] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5396.188653] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5396.190617] ? watchdog+0x40/0x40 [ 5396.191982] ? hrtimer_interrupt+0x128/0x2c0 [ 5396.193916] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5396.196791] ? apic_timer_interrupt+0xf/0x20 [ 5396.198910] [ 5396.199716] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5396.202408] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5396.204654] do_raw_spin_lock+0x177/0x190 [ 5396.206597] _raw_spin_lock+0x21/0x30 [ 5396.208047] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5396.210767] ? do_raw_spin_unlock+0x75/0x190 [ 5396.212674] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5396.215436] ? do_wait_intr+0xf0/0xf0 [ 5396.217065] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5396.219388] mdc_reint+0x2d/0x110 [mdc] [ 5396.221368] mdc_unlink+0x658/0x870 [mdc] [ 5396.223618] lmv_unlink+0x34d/0xbd0 [lmv] [ 5396.225035] ll_unlink+0x24d/0x7f0 [lustre] [ 5396.227404] vfs_unlink+0x202/0x330 [ 5396.229747] do_unlinkat+0x2fa/0x460 [ 5396.231314] __x64_sys_unlink+0x37/0x50 [ 5396.233136] do_syscall_64+0xc1/0x440 [ 5396.234370] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5396.237338] RIP: 0033:0x7f83e5f5c10b [ 5396.239014] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5396.250382] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5396.253763] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5396.256926] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5396.260690] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5396.263798] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5396.266412] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5400.056010] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 5400.059521] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5400.060141] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 5400.089138] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5400.089153] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5400.089156] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5400.089168] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5400.089172] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5400.089178] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5400.089179] RDX: 0000000000004334 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5400.089182] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5400.089184] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5400.089186] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5400.089189] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5400.089191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5400.089193] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5400.089201] Call Trace: [ 5400.089206] [ 5400.089211] ? show_regs.cold.9+0x22/0x2f [ 5400.089219] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5400.089225] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5400.089228] ? watchdog+0x40/0x40 [ 5400.089232] ? hrtimer_interrupt+0x128/0x2c0 [ 5400.089235] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5400.089242] ? apic_timer_interrupt+0xf/0x20 [ 5400.089246] [ 5400.089249] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5400.089253] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5400.089257] do_raw_spin_lock+0x177/0x190 [ 5400.089261] _raw_spin_lock+0x21/0x30 [ 5400.089535] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5400.089915] obd_mod_free+0x46/0x140 [ptlrpc] [ 5400.100817] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5400.100925] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5400.100932] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5400.100938] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5400.107648] mdc_commit_open+0x91/0xb0 [mdc] [ 5400.107700] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5400.107973] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5400.108183] after_reply+0xe2c/0x1780 [ptlrpc] [ 5400.108613] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5400.108829] ? __next_timer_interrupt+0x160/0x160 [ 5400.108837] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5400.130515] [ 5400.130523] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5400.130539] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5400.130543] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5400.134224] ? do_wait_intr+0xf0/0xf0 [ 5400.134247] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5400.134752] kthread+0x1d1/0x200 [ 5400.134759] ? set_kthread_struct+0x70/0x70 [ 5400.134763] ret_from_fork+0x1f/0x30 [ 5400.309027] RAX: 0000000000007c13 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5400.311676] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5400.314625] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5400.317919] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5400.321302] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5400.324417] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5400.327335] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5400.329943] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5400.332489] Call Trace: [ 5400.333613] [ 5400.334329] ? show_regs.cold.9+0x22/0x2f [ 5400.336257] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5400.337993] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5400.339528] ? watchdog+0x40/0x40 [ 5400.340572] ? hrtimer_interrupt+0x128/0x2c0 [ 5400.342251] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5400.344186] ? apic_timer_interrupt+0xf/0x20 [ 5400.346977] [ 5400.347712] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5400.350039] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5400.353121] do_raw_spin_lock+0x177/0x190 [ 5400.354830] _raw_spin_lock+0x21/0x30 [ 5400.356515] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5400.359543] process_one_work+0x2c8/0x700 [ 5400.361023] worker_thread+0x296/0x6e0 [ 5400.362281] ? rescuer_thread+0x570/0x570 [ 5400.363500] kthread+0x1d1/0x200 [ 5400.364487] ? set_kthread_struct+0x70/0x70 [ 5400.366171] ret_from_fork+0x1f/0x30 [ 5405.668522] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2892s! [ 5405.676561] Showing busy workqueues and worker pools: [ 5405.681920] workqueue events: flags=0x0 [ 5405.686245] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5405.693174] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5405.702256] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5405.708404] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5405.712467] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5405.718331] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5405.727579] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5405.735110] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5405.740068] workqueue events_long: flags=0x0 [ 5405.744020] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5405.751071] pending: lu_objects_flush [obdclass] [ 5405.756011] workqueue events_unbound: flags=0x2 [ 5405.759924] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5405.766261] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5405.772609] workqueue events_power_efficient: flags=0x80 [ 5405.775287] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5405.781362] pending: neigh_periodic_work, do_cache_clean [ 5405.785001] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5405.789995] pending: neigh_periodic_work, check_lifetime [ 5405.794664] workqueue mm_percpu_wq: flags=0x8 [ 5405.798280] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5405.811868] pending: vmstat_update, lru_add_drain_per_cpu [ 5405.824151] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5405.831706] pending: vmstat_update [ 5405.833593] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5405.839666] pending: lru_add_drain_per_cpu BAR(48) [ 5405.841444] workqueue ipv6_addrconf: flags=0x40008 [ 5405.844821] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5405.848332] pending: addrconf_verify_work [ 5405.852069] workqueue knbd0-recv: flags=0x1a [ 5405.855466] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5405.859341] in-flight: 650:recv_work [ 5405.860764] workqueue ptlrpc_pinger: flags=0x2 [ 5405.863802] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5405.870648] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5405.880012] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2892s workers=4 idle: 36 466 29757 [ 5405.890237] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5395s workers=2 idle: 652 [ 5408.061876] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 5408.065283] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5408.085686] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5408.090525] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5408.094484] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5408.097396] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5408.121571] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5408.129123] RAX: 0000000000007e8e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5408.136610] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5408.148972] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5408.155905] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5408.161737] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5408.170031] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5408.177627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5408.181895] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5408.185942] Call Trace: [ 5408.190995] [ 5408.191758] ? show_regs.cold.9+0x22/0x2f [ 5408.193177] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5408.197357] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5408.199061] ? watchdog+0x40/0x40 [ 5408.201021] ? hrtimer_interrupt+0x128/0x2c0 [ 5408.206981] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5408.209092] ? apic_timer_interrupt+0xf/0x20 [ 5408.214095] [ 5408.215075] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5408.217311] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5408.223643] do_raw_spin_lock+0x177/0x190 [ 5408.235174] _raw_spin_lock+0x21/0x30 [ 5408.239288] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5408.242321] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5408.247928] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5408.252749] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5408.257088] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5408.267672] ? woken_wake_function+0x30/0x30 [ 5408.270014] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5408.278183] kthread+0x1d1/0x200 [ 5408.279175] ? set_kthread_struct+0x70/0x70 [ 5408.280997] ret_from_fork+0x1f/0x30 [ 5424.062413] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5424.071001] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5424.117677] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5424.129895] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5424.134761] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5424.136944] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5424.153584] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5424.158931] RAX: 0000000000004814 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5424.163922] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5424.167852] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5424.176523] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5424.181390] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5424.186477] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5424.193454] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5424.201303] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5424.207915] Call Trace: [ 5424.209796] [ 5424.211125] ? show_regs.cold.9+0x22/0x2f [ 5424.213822] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5424.216668] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5424.218737] ? watchdog+0x40/0x40 [ 5424.220157] ? hrtimer_interrupt+0x128/0x2c0 [ 5424.221879] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5424.224689] ? apic_timer_interrupt+0xf/0x20 [ 5424.226976] [ 5424.227974] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5424.231155] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5424.233822] do_raw_spin_lock+0x177/0x190 [ 5424.235741] _raw_spin_lock+0x21/0x30 [ 5424.236956] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5424.239857] ? do_raw_spin_unlock+0x75/0x190 [ 5424.241883] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5424.243585] ? do_wait_intr+0xf0/0xf0 [ 5424.247829] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5424.250256] mdc_reint+0x2d/0x110 [mdc] [ 5424.251758] mdc_unlink+0x658/0x870 [mdc] [ 5424.255032] lmv_unlink+0x34d/0xbd0 [lmv] [ 5424.257444] ll_unlink+0x24d/0x7f0 [lustre] [ 5424.260968] vfs_unlink+0x202/0x330 [ 5424.261882] do_unlinkat+0x2fa/0x460 [ 5424.263078] __x64_sys_unlink+0x37/0x50 [ 5424.265806] do_syscall_64+0xc1/0x440 [ 5424.267119] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5424.273520] RIP: 0033:0x7f83e5f5c10b [ 5424.277467] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5424.297023] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5424.307523] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5424.311973] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5424.319735] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5424.326929] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5424.331898] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5428.060533] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 5428.064812] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5428.133918] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5428.144610] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5428.150434] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5428.162231] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5428.165675] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5428.185653] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5428.191929] RAX: 0000000000003a79 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5428.206116] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5428.216624] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5428.221322] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5428.226414] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5428.238880] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5428.254112] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5428.265335] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5428.276220] Call Trace: [ 5428.278774] [ 5428.283963] ? show_regs.cold.9+0x22/0x2f [ 5428.287459] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5428.292406] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5428.296158] ? watchdog+0x40/0x40 [ 5428.298244] ? hrtimer_interrupt+0x128/0x2c0 [ 5428.306705] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5428.309491] ? apic_timer_interrupt+0xf/0x20 [ 5428.314531] [ 5428.317456] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5428.324928] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5428.336255] do_raw_spin_lock+0x177/0x190 [ 5428.341453] _raw_spin_lock+0x21/0x30 [ 5428.344781] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5428.353862] process_one_work+0x2c8/0x700 [ 5428.356684] worker_thread+0x296/0x6e0 [ 5428.360135] ? rescuer_thread+0x570/0x570 [ 5428.365522] kthread+0x1d1/0x200 [ 5428.366742] ? set_kthread_struct+0x70/0x70 [ 5428.379614] ret_from_fork+0x1f/0x30 [ 5432.056200] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 5432.071131] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5432.138826] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5432.150642] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5432.157817] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5432.161101] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5432.170938] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5432.180246] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5432.186929] RDX: 0000000000002b74 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5432.191205] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5432.198835] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5432.207407] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5432.216848] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5432.233400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5432.243985] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5432.253501] Call Trace: [ 5432.254740] [ 5432.259755] ? show_regs.cold.9+0x22/0x2f [ 5432.272212] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5432.280680] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5432.284881] ? watchdog+0x40/0x40 [ 5432.290751] ? hrtimer_interrupt+0x128/0x2c0 [ 5432.296150] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5432.302624] ? apic_timer_interrupt+0xf/0x20 [ 5432.307125] [ 5432.309160] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5432.311922] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5432.318871] do_raw_spin_lock+0x177/0x190 [ 5432.321419] _raw_spin_lock+0x21/0x30 [ 5432.323179] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5432.337675] obd_mod_free+0x46/0x140 [ptlrpc] [ 5432.344395] mdc_commit_open+0x91/0xb0 [mdc] [ 5432.355528] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5432.365168] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5432.375397] after_reply+0xe2c/0x1780 [ptlrpc] [ 5432.381747] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5432.386700] ? __next_timer_interrupt+0x160/0x160 [ 5432.395937] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5432.400740] ? do_wait_intr+0xf0/0xf0 [ 5432.407538] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5432.414101] kthread+0x1d1/0x200 [ 5432.414963] ? set_kthread_struct+0x70/0x70 [ 5432.419239] ret_from_fork+0x1f/0x30 [ 5436.059024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 5436.062799] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5436.081080] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5436.085361] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5436.089173] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5436.090733] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5436.096215] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5436.098327] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5436.101636] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5436.105894] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5436.109138] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5436.111776] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5436.113699] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5436.116692] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5436.118983] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5436.121926] Call Trace: [ 5436.122936] [ 5436.123877] ? show_regs.cold.9+0x22/0x2f [ 5436.125588] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5436.127737] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5436.129365] ? watchdog+0x40/0x40 [ 5436.135596] ? hrtimer_interrupt+0x128/0x2c0 [ 5436.137321] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5436.139412] ? apic_timer_interrupt+0xf/0x20 [ 5436.141213] [ 5436.142128] ? native_safe_halt+0x2e/0x50 [ 5436.144044] kvm_wait+0xcd/0xe0 [ 5436.145243] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5436.147529] do_raw_spin_lock+0x177/0x190 [ 5436.149195] _raw_spin_lock+0x21/0x30 [ 5436.150390] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5436.152401] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5436.154817] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5436.156778] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5436.158846] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5436.160591] ? woken_wake_function+0x30/0x30 [ 5436.161709] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5436.163225] kthread+0x1d1/0x200 [ 5436.166731] ? set_kthread_struct+0x70/0x70 [ 5436.168571] ret_from_fork+0x1f/0x30 [ 5436.385028] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2923s! [ 5436.388746] Showing busy workqueues and worker pools: [ 5436.391070] workqueue events: flags=0x0 [ 5436.392982] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5436.395697] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5436.399767] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5436.402743] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5436.405383] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5436.408526] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5436.411937] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5436.415294] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5436.419081] workqueue events_long: flags=0x0 [ 5436.421039] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5436.423664] pending: lu_objects_flush [obdclass] [ 5436.426262] workqueue events_unbound: flags=0x2 [ 5436.428088] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5436.431351] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5436.434320] workqueue events_power_efficient: flags=0x80 [ 5436.437019] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5436.439590] pending: neigh_periodic_work, do_cache_clean [ 5436.441894] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5436.445586] pending: neigh_periodic_work, check_lifetime [ 5436.448876] workqueue mm_percpu_wq: flags=0x8 [ 5436.451090] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5436.454882] pending: vmstat_update, lru_add_drain_per_cpu [ 5436.456812] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5436.459793] pending: vmstat_update [ 5436.461528] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5436.464362] pending: lru_add_drain_per_cpu BAR(48) [ 5436.466812] workqueue ipv6_addrconf: flags=0x40008 [ 5436.469071] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5436.472003] pending: addrconf_verify_work [ 5436.473974] workqueue knbd0-recv: flags=0x1a [ 5436.476535] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5436.479653] in-flight: 650:recv_work [ 5436.480850] workqueue ptlrpc_pinger: flags=0x2 [ 5436.483372] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5436.487246] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5436.490377] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2923s workers=4 idle: 36 466 29757 [ 5436.494607] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5426s workers=2 idle: 652 [ 5452.071373] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5452.078887] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5452.149748] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5452.160510] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5452.167204] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5452.178854] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5452.221462] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5452.229675] RAX: 0000000000005087 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5452.245189] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5452.259507] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5452.274761] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5452.283940] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5452.285783] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5452.290504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5452.293702] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5452.297212] Call Trace: [ 5452.298423] [ 5452.299398] ? show_regs.cold.9+0x22/0x2f [ 5452.301448] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5452.304132] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5452.306195] ? watchdog+0x40/0x40 [ 5452.307777] ? hrtimer_interrupt+0x128/0x2c0 [ 5452.310405] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5452.312733] ? apic_timer_interrupt+0xf/0x20 [ 5452.314904] [ 5452.315997] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5452.318450] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5452.320882] do_raw_spin_lock+0x177/0x190 [ 5452.322870] _raw_spin_lock+0x21/0x30 [ 5452.324881] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5452.334717] ? do_raw_spin_unlock+0x75/0x190 [ 5452.344627] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5452.346606] ? do_wait_intr+0xf0/0xf0 [ 5452.355428] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5452.361006] mdc_reint+0x2d/0x110 [mdc] [ 5452.365906] mdc_unlink+0x658/0x870 [mdc] [ 5452.371229] lmv_unlink+0x34d/0xbd0 [lmv] [ 5452.373608] ll_unlink+0x24d/0x7f0 [lustre] [ 5452.375680] vfs_unlink+0x202/0x330 [ 5452.377465] do_unlinkat+0x2fa/0x460 [ 5452.378879] __x64_sys_unlink+0x37/0x50 [ 5452.384555] do_syscall_64+0xc1/0x440 [ 5452.385923] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5452.395775] RIP: 0033:0x7f83e5f5c10b [ 5452.399241] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5452.417338] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5452.422384] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5452.429848] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5452.436764] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5452.445861] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5452.457520] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5456.060188] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5456.065177] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5456.107374] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5456.121128] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5456.135666] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5456.144928] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5456.150781] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5456.177709] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5456.184943] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5456.194663] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5456.198722] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5456.206487] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5456.213475] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5456.222421] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5456.225491] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5456.233716] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5456.240987] Call Trace: [ 5456.242496] [ 5456.244173] ? show_regs.cold.9+0x22/0x2f [ 5456.246197] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5456.248479] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5456.253792] ? watchdog+0x40/0x40 [ 5456.256820] ? hrtimer_interrupt+0x128/0x2c0 [ 5456.264874] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5456.267892] ? apic_timer_interrupt+0xf/0x20 [ 5456.271382] [ 5456.275677] ? native_safe_halt+0x2e/0x50 [ 5456.281687] kvm_wait+0xcd/0xe0 [ 5456.282946] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5456.290505] do_raw_spin_lock+0x177/0x190 [ 5456.295754] _raw_spin_lock+0x21/0x30 [ 5456.300748] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5456.308733] process_one_work+0x2c8/0x700 [ 5456.311835] worker_thread+0x296/0x6e0 [ 5456.317338] ? rescuer_thread+0x570/0x570 [ 5456.322685] kthread+0x1d1/0x200 [ 5456.332283] ? set_kthread_struct+0x70/0x70 [ 5456.337555] ret_from_fork+0x1f/0x30 [ 5461.036548] rcu: INFO: rcu_sched self-detected stall on CPU [ 5461.039822] rcu: 0-....: (2192987 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 5461.054494] (t=3480095 jiffies g=262441 q=58146) [ 5461.059811] rcu: rcu_sched kthread starved for 2949287 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 5461.075499] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5461.085882] rcu: RCU grace-period kthread stack dump: [ 5461.087562] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5461.112968] Call Trace: [ 5461.114373] __schedule+0x351/0xcb0 [ 5461.116511] schedule+0xc0/0x180 [ 5461.121104] schedule_timeout+0xb4/0x190 [ 5461.124558] ? do_raw_spin_unlock+0x75/0x190 [ 5461.129819] ? __next_timer_interrupt+0x160/0x160 [ 5461.140139] ? prepare_to_swait_event+0xf3/0x140 [ 5461.144951] rcu_gp_kthread+0x8dd/0x15b0 [ 5461.155167] ? force_qs_rnp+0x2c0/0x2c0 [ 5461.159014] kthread+0x1d1/0x200 [ 5461.167479] ? set_kthread_struct+0x70/0x70 [ 5461.174863] ret_from_fork+0x1f/0x30 [ 5461.184301] rcu: Stack dump where RCU GP kthread last ran: [ 5461.186730] Sending NMI from CPU 0 to CPUs 2: [ 5461.195673] NMI backtrace for cpu 2 [ 5461.195680] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5461.195683] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5461.195685] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5461.195687] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5461.195689] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5461.195691] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 [ 5461.195692] RAX: 000000000000272a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5461.195694] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5461.195695] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5461.195697] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5461.195698] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5461.195700] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5461.195701] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5461.195702] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5461.195704] Call Trace: [ 5461.195704] [ 5461.195705] ? show_regs.cold.9+0x22/0x2f [ 5461.195706] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 5461.195707] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 5461.195708] ? nmi_handle+0xbd/0x210 [ 5461.195709] ? default_do_nmi+0x4b/0x1b0 [ 5461.195710] ? do_nmi+0x280/0x380 [ 5461.195711] ? end_repeat_nmi+0x16/0x1f [ 5461.195712] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5461.195714] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5461.195715] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5461.195716] [ 5461.195717] do_raw_spin_lock+0x177/0x190 [ 5461.195718] _raw_spin_lock+0x21/0x30 [ 5461.195719] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5461.195720] process_one_work+0x2c8/0x700 [ 5461.195721] worker_thread+0x296/0x6e0 [ 5461.195722] ? rescuer_thread+0x570/0x570 [ 5461.195723] kthread+0x1d1/0x200 [ 5461.195724] ? set_kthread_struct+0x70/0x70 [ 5461.195725] ret_from_fork+0x1f/0x30 [ 5461.195797] NMI backtrace for cpu 0 [ 5461.358621] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5461.365608] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5461.370908] Call Trace: [ 5461.372277] [ 5461.374695] ? dump_stack+0xbb/0x10e [ 5461.376886] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5461.380954] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5461.385243] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5461.387948] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5461.393638] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5461.396311] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5461.400662] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5461.404936] ? update_process_times+0x6d/0xc0 [ 5461.408700] ? tick_sched_handle.isra.18+0x27/0x90 [ 5461.411027] ? tick_sched_timer+0x53/0xb0 [ 5461.413453] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5461.417456] ? tick_sched_handle.isra.18+0x90/0x90 [ 5461.420651] ? hrtimer_interrupt+0x128/0x2c0 [ 5461.422348] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5461.424365] ? apic_timer_interrupt+0xf/0x20 [ 5461.427288] [ 5461.428428] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5461.431313] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5461.435421] ? do_raw_spin_lock+0x177/0x190 [ 5461.440277] ? _raw_spin_lock+0x21/0x30 [ 5461.441921] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5461.447463] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5461.451876] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5461.455467] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5461.459624] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5461.464612] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5461.467379] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5461.473419] ? __next_timer_interrupt+0x160/0x160 [ 5461.475834] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5461.482397] ? do_wait_intr+0xf0/0xf0 [ 5461.486495] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5461.492471] ? kthread+0x1d1/0x200 [ 5461.495948] ? set_kthread_struct+0x70/0x70 [ 5461.498764] ? ret_from_fork+0x1f/0x30 [ 5464.059011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 5464.066886] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5464.093135] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5464.102023] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5464.108915] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5464.113284] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5464.139366] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5464.147747] RAX: 000000000000671e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5464.156249] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5464.169743] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5464.183702] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5464.188199] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5464.198784] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5464.207199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5464.209971] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5464.217979] Call Trace: [ 5464.218884] [ 5464.219830] ? show_regs.cold.9+0x22/0x2f [ 5464.223679] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5464.225905] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5464.227353] ? watchdog+0x40/0x40 [ 5464.232122] ? hrtimer_interrupt+0x128/0x2c0 [ 5464.235488] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5464.238775] ? apic_timer_interrupt+0xf/0x20 [ 5464.241836] [ 5464.242501] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5464.246109] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5464.250506] do_raw_spin_lock+0x177/0x190 [ 5464.253286] _raw_spin_lock+0x21/0x30 [ 5464.254558] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5464.267796] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5464.278368] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5464.282058] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5464.286876] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5464.294230] ? woken_wake_function+0x30/0x30 [ 5464.295716] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5464.304638] kthread+0x1d1/0x200 [ 5464.308815] ? set_kthread_struct+0x70/0x70 [ 5464.312494] ret_from_fork+0x1f/0x30 [ 5466.603037] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2953s! [ 5466.606122] Showing busy workqueues and worker pools: [ 5466.608401] workqueue events: flags=0x0 [ 5466.612654] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5466.618400] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5466.632404] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5466.636235] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5466.644373] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5466.649937] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5466.658110] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5466.668382] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5466.676397] workqueue events_long: flags=0x0 [ 5466.681179] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5466.684333] pending: lu_objects_flush [obdclass] [ 5466.687311] workqueue events_unbound: flags=0x2 [ 5466.691525] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5466.696490] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5466.700614] workqueue events_power_efficient: flags=0x80 [ 5466.703943] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5466.710414] pending: neigh_periodic_work, do_cache_clean [ 5466.713257] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5466.716547] pending: neigh_periodic_work, check_lifetime [ 5466.719374] workqueue mm_percpu_wq: flags=0x8 [ 5466.724981] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5466.734877] pending: vmstat_update, lru_add_drain_per_cpu [ 5466.741664] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5466.747086] pending: vmstat_update [ 5466.749311] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5466.754033] pending: lru_add_drain_per_cpu BAR(48) [ 5466.758673] workqueue ipv6_addrconf: flags=0x40008 [ 5466.763795] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5466.770539] pending: addrconf_verify_work [ 5466.775783] workqueue knbd0-recv: flags=0x1a [ 5466.777325] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5466.789999] in-flight: 650:recv_work [ 5466.795907] workqueue ptlrpc_pinger: flags=0x2 [ 5466.798834] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5466.807486] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5466.812075] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2953s workers=4 idle: 36 466 29757 [ 5466.818054] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5456s workers=2 idle: 652 [ 5480.068266] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5480.081536] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5480.156974] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5480.170655] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5480.184075] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5480.191184] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5480.216605] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5480.230899] RAX: 0000000000007b08 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5480.240475] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5480.250518] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5480.264374] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5480.273782] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5480.283925] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5480.298909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5480.303295] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5480.312662] Call Trace: [ 5480.313697] [ 5480.318379] ? show_regs.cold.9+0x22/0x2f [ 5480.322611] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5480.328245] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5480.329979] ? watchdog+0x40/0x40 [ 5480.332645] ? hrtimer_interrupt+0x128/0x2c0 [ 5480.334537] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5480.335834] ? apic_timer_interrupt+0xf/0x20 [ 5480.337957] [ 5480.339155] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5480.341310] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5480.343801] do_raw_spin_lock+0x177/0x190 [ 5480.345436] _raw_spin_lock+0x21/0x30 [ 5480.347281] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5480.349528] ? do_raw_spin_unlock+0x75/0x190 [ 5480.351193] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5480.353297] ? do_wait_intr+0xf0/0xf0 [ 5480.355120] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5480.357805] mdc_reint+0x2d/0x110 [mdc] [ 5480.359759] mdc_unlink+0x658/0x870 [mdc] [ 5480.361001] lmv_unlink+0x34d/0xbd0 [lmv] [ 5480.362007] ll_unlink+0x24d/0x7f0 [lustre] [ 5480.363320] vfs_unlink+0x202/0x330 [ 5480.364210] do_unlinkat+0x2fa/0x460 [ 5480.365223] __x64_sys_unlink+0x37/0x50 [ 5480.366414] do_syscall_64+0xc1/0x440 [ 5480.367928] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5480.369499] RIP: 0033:0x7f83e5f5c10b [ 5480.370546] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5480.389866] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5480.393565] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5480.398108] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5480.401840] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5480.408704] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5480.415400] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5488.060595] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 5488.067170] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5488.114672] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5488.121622] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5488.128836] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5488.134887] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5488.140041] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5488.158056] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5488.164263] RAX: 0000000000003d93 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5488.171505] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5488.187628] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5488.206314] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5488.217269] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5488.237040] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5488.245879] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5488.254530] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5488.269843] Call Trace: [ 5488.270633] [ 5488.271267] ? show_regs.cold.9+0x22/0x2f [ 5488.286509] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5488.292498] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5488.301485] ? watchdog+0x40/0x40 [ 5488.302562] ? hrtimer_interrupt+0x128/0x2c0 [ 5488.311315] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5488.315254] ? apic_timer_interrupt+0xf/0x20 [ 5488.316498] [ 5488.322162] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5488.324270] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5488.326360] do_raw_spin_lock+0x177/0x190 [ 5488.328658] _raw_spin_lock+0x21/0x30 [ 5488.330799] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5488.333466] process_one_work+0x2c8/0x700 [ 5488.335366] worker_thread+0x296/0x6e0 [ 5488.344230] ? rescuer_thread+0x570/0x570 [ 5488.347250] kthread+0x1d1/0x200 [ 5488.351471] ? set_kthread_struct+0x70/0x70 [ 5488.352737] ret_from_fork+0x1f/0x30 [ 5492.059025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 5492.062158] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 5492.062484] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5492.062659] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5492.062665] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5492.062668] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5492.062688] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5492.062692] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5492.062696] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5492.062699] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5492.062701] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5492.062703] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5492.062706] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5492.062709] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5492.062712] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5492.062715] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5492.062722] Call Trace: [ 5492.062727] [ 5492.062732] ? show_regs.cold.9+0x22/0x2f [ 5492.062740] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5492.062746] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5492.062750] ? watchdog+0x40/0x40 [ 5492.062754] ? hrtimer_interrupt+0x128/0x2c0 [ 5492.062759] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5492.062765] ? apic_timer_interrupt+0xf/0x20 [ 5492.062770] [ 5492.062773] ? native_safe_halt+0x2e/0x50 [ 5492.062777] kvm_wait+0xcd/0xe0 [ 5492.062783] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5492.062792] do_raw_spin_lock+0x177/0x190 [ 5492.062796] _raw_spin_lock+0x21/0x30 [ 5492.062800] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5492.063247] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5492.063450] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5492.063802] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5492.064212] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5492.064579] ? woken_wake_function+0x30/0x30 [ 5492.064587] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5492.065210] kthread+0x1d1/0x200 [ 5492.065216] ? set_kthread_struct+0x70/0x70 [ 5492.065219] ret_from_fork+0x1f/0x30 [ 5492.322408] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5492.386151] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5492.396220] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5492.408540] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5492.420081] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5492.435519] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5492.452518] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5492.458092] RDX: 0000000000007dec RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5492.467701] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5492.477373] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5492.479556] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5492.492064] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5492.504225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5492.511926] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5492.517434] Call Trace: [ 5492.518136] [ 5492.518729] ? show_regs.cold.9+0x22/0x2f [ 5492.523975] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5492.525743] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5492.529189] ? watchdog+0x40/0x40 [ 5492.533047] ? hrtimer_interrupt+0x128/0x2c0 [ 5492.540522] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5492.546328] ? apic_timer_interrupt+0xf/0x20 [ 5492.547591] [ 5492.548260] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5492.554068] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5492.560428] do_raw_spin_lock+0x177/0x190 [ 5492.563967] _raw_spin_lock+0x21/0x30 [ 5492.567208] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5492.569687] obd_mod_free+0x46/0x140 [ptlrpc] [ 5492.572038] mdc_commit_open+0x91/0xb0 [mdc] [ 5492.574173] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5492.576487] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5492.578775] after_reply+0xe2c/0x1780 [ptlrpc] [ 5492.582022] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5492.584305] ? __next_timer_interrupt+0x160/0x160 [ 5492.586121] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5492.588206] ? do_wait_intr+0xf0/0xf0 [ 5492.589571] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5492.591644] kthread+0x1d1/0x200 [ 5492.593011] ? set_kthread_struct+0x70/0x70 [ 5492.594684] ret_from_fork+0x1f/0x30 [ 5497.318670] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2984s! [ 5497.325065] Showing busy workqueues and worker pools: [ 5497.327677] workqueue events: flags=0x0 [ 5497.329421] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5497.338265] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5497.350208] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5497.355918] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5497.361220] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5497.364394] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5497.370996] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5497.379498] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5497.385136] workqueue events_long: flags=0x0 [ 5497.387125] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5497.391461] pending: lu_objects_flush [obdclass] [ 5497.399492] workqueue events_unbound: flags=0x2 [ 5497.402319] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5497.410091] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5497.412906] workqueue events_power_efficient: flags=0x80 [ 5497.417523] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5497.423286] pending: neigh_periodic_work, do_cache_clean [ 5497.427377] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5497.434443] pending: neigh_periodic_work, check_lifetime [ 5497.436415] workqueue mm_percpu_wq: flags=0x8 [ 5497.445031] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5497.456725] pending: vmstat_update, lru_add_drain_per_cpu [ 5497.459252] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5497.463460] pending: vmstat_update [ 5497.465252] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5497.468554] pending: lru_add_drain_per_cpu BAR(48) [ 5497.472330] workqueue ipv6_addrconf: flags=0x40008 [ 5497.474373] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5497.479166] pending: addrconf_verify_work [ 5497.480427] workqueue knbd0-recv: flags=0x1a [ 5497.483417] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5497.490255] in-flight: 650:recv_work [ 5497.497810] workqueue ptlrpc_pinger: flags=0x2 [ 5497.506392] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5497.516172] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5497.519243] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2984s workers=4 idle: 36 466 29757 [ 5497.522279] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5487s workers=2 idle: 652 [ 5508.062034] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5508.065971] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5508.104411] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5508.112571] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5508.116177] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5508.117928] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5508.125584] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5508.128033] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5508.131031] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5508.134145] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5508.137539] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5508.141022] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5508.144116] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5508.147552] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5508.149997] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5508.153139] Call Trace: [ 5508.154143] [ 5508.154876] ? show_regs.cold.9+0x22/0x2f [ 5508.156558] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5508.158614] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5508.160491] ? watchdog+0x40/0x40 [ 5508.161970] ? hrtimer_interrupt+0x128/0x2c0 [ 5508.163772] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5508.166056] ? apic_timer_interrupt+0xf/0x20 [ 5508.167833] [ 5508.168712] ? native_safe_halt+0x2e/0x50 [ 5508.170314] kvm_wait+0xcd/0xe0 [ 5508.171430] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5508.172867] do_raw_spin_lock+0x177/0x190 [ 5508.174034] _raw_spin_lock+0x21/0x30 [ 5508.175385] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5508.177315] ? do_raw_spin_unlock+0x75/0x190 [ 5508.178502] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5508.180793] ? do_wait_intr+0xf0/0xf0 [ 5508.182506] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5508.185053] mdc_reint+0x2d/0x110 [mdc] [ 5508.186489] mdc_unlink+0x658/0x870 [mdc] [ 5508.187982] lmv_unlink+0x34d/0xbd0 [lmv] [ 5508.189155] ll_unlink+0x24d/0x7f0 [lustre] [ 5508.190669] vfs_unlink+0x202/0x330 [ 5508.191773] do_unlinkat+0x2fa/0x460 [ 5508.192847] __x64_sys_unlink+0x37/0x50 [ 5508.193957] do_syscall_64+0xc1/0x440 [ 5508.195169] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5508.196702] RIP: 0033:0x7f83e5f5c10b [ 5508.197773] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5508.205643] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5508.208961] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5508.211986] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5508.214979] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5508.218108] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5508.221169] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5516.060028] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 5516.064213] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5516.099880] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5516.111759] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5516.117461] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5516.121758] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5516.127658] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5516.135214] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5516.141099] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5516.144121] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5516.147811] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5516.151419] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5516.154952] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5516.158047] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5516.161613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5516.163222] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5516.168096] Call Trace: [ 5516.169996] [ 5516.170765] ? show_regs.cold.9+0x22/0x2f [ 5516.172115] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5516.175149] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5516.178630] ? watchdog+0x40/0x40 [ 5516.180991] ? hrtimer_interrupt+0x128/0x2c0 [ 5516.184688] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5516.187740] ? apic_timer_interrupt+0xf/0x20 [ 5516.189799] [ 5516.190845] ? native_safe_halt+0x2e/0x50 [ 5516.192789] kvm_wait+0xcd/0xe0 [ 5516.194177] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5516.196330] do_raw_spin_lock+0x177/0x190 [ 5516.200718] _raw_spin_lock+0x21/0x30 [ 5516.202217] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5516.205422] process_one_work+0x2c8/0x700 [ 5516.206599] worker_thread+0x296/0x6e0 [ 5516.207767] ? rescuer_thread+0x570/0x570 [ 5516.210180] kthread+0x1d1/0x200 [ 5516.211868] ? set_kthread_struct+0x70/0x70 [ 5516.214190] ret_from_fork+0x1f/0x30 [ 5520.059017] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 5520.061242] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5520.092365] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5520.098247] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5520.102737] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5520.105287] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5520.113962] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5520.117307] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5520.120512] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5520.123999] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5520.127218] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5520.130676] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5520.133946] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5520.136946] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5520.139418] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5520.142630] Call Trace: [ 5520.143722] [ 5520.144948] ? show_regs.cold.9+0x22/0x2f [ 5520.147020] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5520.149228] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5520.151520] ? watchdog+0x40/0x40 [ 5520.153418] ? hrtimer_interrupt+0x128/0x2c0 [ 5520.155585] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5520.157696] ? apic_timer_interrupt+0xf/0x20 [ 5520.159261] [ 5520.160022] ? native_safe_halt+0x2e/0x50 [ 5520.161860] kvm_wait+0xcd/0xe0 [ 5520.163313] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5520.165497] do_raw_spin_lock+0x177/0x190 [ 5520.167177] _raw_spin_lock+0x21/0x30 [ 5520.168716] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5520.171494] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5520.173833] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5520.176359] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5520.178798] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5520.181144] ? woken_wake_function+0x30/0x30 [ 5520.183045] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5520.185968] kthread+0x1d1/0x200 [ 5520.187927] ? set_kthread_struct+0x70/0x70 [ 5520.189772] ret_from_fork+0x1f/0x30 [ 5524.056068] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 5524.059247] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5524.097736] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5524.104282] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5524.112694] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5524.115449] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5524.126849] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5524.129321] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5524.132768] RDX: 0000000000002ab4 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5524.135752] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5524.138763] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5524.141933] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5524.150653] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5524.161501] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5524.167931] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5524.172294] Call Trace: [ 5524.174724] [ 5524.175549] ? show_regs.cold.9+0x22/0x2f [ 5524.179888] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5524.183094] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5524.185128] ? watchdog+0x40/0x40 [ 5524.187043] ? hrtimer_interrupt+0x128/0x2c0 [ 5524.188982] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5524.191155] ? apic_timer_interrupt+0xf/0x20 [ 5524.193517] [ 5524.194341] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5524.200932] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5524.202930] do_raw_spin_lock+0x177/0x190 [ 5524.208983] _raw_spin_lock+0x21/0x30 [ 5524.210022] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5524.213451] obd_mod_free+0x46/0x140 [ptlrpc] [ 5524.217261] mdc_commit_open+0x91/0xb0 [mdc] [ 5524.219793] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5524.222539] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5524.228969] after_reply+0xe2c/0x1780 [ptlrpc] [ 5524.234145] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5524.236193] ? __next_timer_interrupt+0x160/0x160 [ 5524.241399] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5524.253067] ? do_wait_intr+0xf0/0xf0 [ 5524.255099] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5524.260098] kthread+0x1d1/0x200 [ 5524.261690] ? set_kthread_struct+0x70/0x70 [ 5524.266220] ret_from_fork+0x1f/0x30 [ 5528.036043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3014s! [ 5528.039628] Showing busy workqueues and worker pools: [ 5528.042179] workqueue events: flags=0x0 [ 5528.044017] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5528.047711] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5528.051264] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5528.054334] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5528.057060] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5528.060021] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5528.063168] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5528.066250] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5528.070619] workqueue events_long: flags=0x0 [ 5528.072371] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5528.075280] pending: lu_objects_flush [obdclass] [ 5528.077740] workqueue events_unbound: flags=0x2 [ 5528.079449] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5528.082337] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5528.085060] workqueue events_power_efficient: flags=0x80 [ 5528.087916] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5528.090850] pending: neigh_periodic_work, do_cache_clean [ 5528.092962] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5528.094943] pending: neigh_periodic_work, check_lifetime [ 5528.097589] workqueue mm_percpu_wq: flags=0x8 [ 5528.099379] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5528.102665] pending: vmstat_update, lru_add_drain_per_cpu [ 5528.105258] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5528.108158] pending: vmstat_update [ 5528.109950] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5528.113040] pending: lru_add_drain_per_cpu BAR(48) [ 5528.115427] workqueue ipv6_addrconf: flags=0x40008 [ 5528.117713] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5528.121407] pending: addrconf_verify_work [ 5528.123358] workqueue knbd0-recv: flags=0x1a [ 5528.125251] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5528.128130] in-flight: 650:recv_work [ 5528.129630] workqueue ptlrpc_pinger: flags=0x2 [ 5528.131599] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5528.134607] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5528.137298] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3014s workers=4 idle: 36 466 29757 [ 5528.141343] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5518s workers=2 idle: 652 [ 5536.062693] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5536.070539] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5536.096033] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5536.102647] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5536.107000] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5536.110297] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5536.118747] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5536.122373] RAX: 0000000000007f85 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5536.130136] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5536.136895] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5536.144552] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5536.152759] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5536.163940] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5536.168116] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5536.170967] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5536.174089] Call Trace: [ 5536.175027] [ 5536.175667] ? show_regs.cold.9+0x22/0x2f [ 5536.177377] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5536.179469] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5536.181417] ? watchdog+0x40/0x40 [ 5536.182870] ? hrtimer_interrupt+0x128/0x2c0 [ 5536.184975] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5536.186422] ? apic_timer_interrupt+0xf/0x20 [ 5536.188626] [ 5536.189706] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5536.191895] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5536.193200] do_raw_spin_lock+0x177/0x190 [ 5536.194190] _raw_spin_lock+0x21/0x30 [ 5536.195680] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5536.200253] ? do_raw_spin_unlock+0x75/0x190 [ 5536.202200] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5536.203942] ? do_wait_intr+0xf0/0xf0 [ 5536.206490] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5536.210165] mdc_reint+0x2d/0x110 [mdc] [ 5536.211424] mdc_unlink+0x658/0x870 [mdc] [ 5536.212648] lmv_unlink+0x34d/0xbd0 [lmv] [ 5536.213749] ll_unlink+0x24d/0x7f0 [lustre] [ 5536.215713] vfs_unlink+0x202/0x330 [ 5536.216886] do_unlinkat+0x2fa/0x460 [ 5536.218503] __x64_sys_unlink+0x37/0x50 [ 5536.219628] do_syscall_64+0xc1/0x440 [ 5536.220789] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5536.223352] RIP: 0033:0x7f83e5f5c10b [ 5536.224646] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5536.234611] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5536.238317] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5536.241919] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5536.245207] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5536.248957] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5536.252696] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5544.061940] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 5544.073004] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5544.123540] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5544.132678] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5544.140927] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5544.146268] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5544.151341] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5544.163575] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5544.167033] RAX: 0000000000006d07 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5544.182562] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5544.189477] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5544.194602] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5544.202087] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5544.207787] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5544.215180] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5544.223876] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5544.231403] Call Trace: [ 5544.234504] [ 5544.236131] ? show_regs.cold.9+0x22/0x2f [ 5544.237908] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5544.240879] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5544.256778] ? watchdog+0x40/0x40 [ 5544.260702] ? hrtimer_interrupt+0x128/0x2c0 [ 5544.270828] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5544.276670] ? apic_timer_interrupt+0xf/0x20 [ 5544.278369] [ 5544.279138] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5544.285553] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5544.288585] do_raw_spin_lock+0x177/0x190 [ 5544.290746] _raw_spin_lock+0x21/0x30 [ 5544.292680] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5544.300839] process_one_work+0x2c8/0x700 [ 5544.304027] worker_thread+0x296/0x6e0 [ 5544.307396] ? rescuer_thread+0x570/0x570 [ 5544.308910] kthread+0x1d1/0x200 [ 5544.310475] ? set_kthread_struct+0x70/0x70 [ 5544.312282] ret_from_fork+0x1f/0x30 [ 5548.059022] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 5548.064468] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5548.107487] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5548.120732] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5548.126741] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5548.129306] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5548.150209] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5548.157260] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5548.166619] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5548.172147] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5548.176967] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5548.181906] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5548.186871] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5548.189681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5548.192067] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5548.196950] Call Trace: [ 5548.197900] [ 5548.199635] ? show_regs.cold.9+0x22/0x2f [ 5548.202202] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5548.205747] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5548.212977] ? watchdog+0x40/0x40 [ 5548.215701] ? hrtimer_interrupt+0x128/0x2c0 [ 5548.219222] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5548.222855] ? apic_timer_interrupt+0xf/0x20 [ 5548.224320] [ 5548.226790] ? native_safe_halt+0x2e/0x50 [ 5548.228415] kvm_wait+0xcd/0xe0 [ 5548.229942] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5548.231828] do_raw_spin_lock+0x177/0x190 [ 5548.233790] _raw_spin_lock+0x21/0x30 [ 5548.235680] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5548.238716] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5548.241161] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5548.243897] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5548.246122] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5548.248801] ? woken_wake_function+0x30/0x30 [ 5548.250713] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5548.256447] kthread+0x1d1/0x200 [ 5548.258762] ? set_kthread_struct+0x70/0x70 [ 5548.260698] ret_from_fork+0x1f/0x30 [ 5556.056239] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 5556.059692] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5556.116126] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5556.122549] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5556.127628] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5556.129706] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5556.140933] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5556.146713] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5556.150662] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5556.154879] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5556.157763] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5556.165876] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5556.169921] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5556.174341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5556.179454] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5556.182815] Call Trace: [ 5556.184033] [ 5556.184736] ? show_regs.cold.9+0x22/0x2f [ 5556.186525] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5556.189902] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5556.193618] ? watchdog+0x40/0x40 [ 5556.195840] ? hrtimer_interrupt+0x128/0x2c0 [ 5556.199459] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5556.202974] ? apic_timer_interrupt+0xf/0x20 [ 5556.207166] [ 5556.208357] ? native_safe_halt+0x2e/0x50 [ 5556.210140] kvm_wait+0xcd/0xe0 [ 5556.213104] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5556.217307] do_raw_spin_lock+0x177/0x190 [ 5556.221636] _raw_spin_lock+0x21/0x30 [ 5556.225135] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5556.230233] obd_mod_free+0x46/0x140 [ptlrpc] [ 5556.233618] mdc_commit_open+0x91/0xb0 [mdc] [ 5556.234657] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5556.239724] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5556.243180] after_reply+0xe2c/0x1780 [ptlrpc] [ 5556.245776] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5556.248167] ? __next_timer_interrupt+0x160/0x160 [ 5556.251273] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5556.253537] ? do_wait_intr+0xf0/0xf0 [ 5556.255127] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5556.258449] kthread+0x1d1/0x200 [ 5556.259620] ? set_kthread_struct+0x70/0x70 [ 5556.261681] ret_from_fork+0x1f/0x30 [ 5558.241133] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3045s! [ 5558.244216] Showing busy workqueues and worker pools: [ 5558.245946] workqueue events: flags=0x0 [ 5558.247443] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5558.249952] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5558.253443] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5558.255677] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5558.258361] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5558.260978] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5558.263786] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5558.266514] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5558.270094] workqueue events_long: flags=0x0 [ 5558.271494] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5558.273835] pending: lu_objects_flush [obdclass] [ 5558.276123] workqueue events_unbound: flags=0x2 [ 5558.278044] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5558.280598] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5558.283284] workqueue events_power_efficient: flags=0x80 [ 5558.285073] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5558.287577] pending: neigh_periodic_work, do_cache_clean [ 5558.289631] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5558.292288] pending: neigh_periodic_work, check_lifetime [ 5558.294483] workqueue mm_percpu_wq: flags=0x8 [ 5558.296385] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5558.298607] pending: vmstat_update, lru_add_drain_per_cpu [ 5558.300690] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5558.303536] pending: vmstat_update [ 5558.305056] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5558.308230] pending: lru_add_drain_per_cpu BAR(48) [ 5558.310512] workqueue ipv6_addrconf: flags=0x40008 [ 5558.312563] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5558.314970] pending: addrconf_verify_work [ 5558.316558] workqueue knbd0-recv: flags=0x1a [ 5558.318121] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5558.320553] in-flight: 650:recv_work [ 5558.322302] workqueue ptlrpc_pinger: flags=0x2 [ 5558.324425] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5558.327476] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5558.330402] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3045s workers=4 idle: 36 466 29757 [ 5558.333503] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5548s workers=2 idle: 652 [ 5564.066736] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5564.070020] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5564.133425] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5564.146373] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5564.150688] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5564.154389] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5564.173866] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5564.183823] RAX: 0000000000005307 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5564.193231] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5564.203304] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5564.207454] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5564.216476] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5564.220479] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5564.230171] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5564.234739] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5564.242866] Call Trace: [ 5564.243748] [ 5564.244369] ? show_regs.cold.9+0x22/0x2f [ 5564.245657] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5564.252886] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5564.258517] ? watchdog+0x40/0x40 [ 5564.264148] ? hrtimer_interrupt+0x128/0x2c0 [ 5564.268073] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5564.271841] ? apic_timer_interrupt+0xf/0x20 [ 5564.274868] [ 5564.275640] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5564.278477] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5564.283620] do_raw_spin_lock+0x177/0x190 [ 5564.286154] _raw_spin_lock+0x21/0x30 [ 5564.288600] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5564.295134] ? do_raw_spin_unlock+0x75/0x190 [ 5564.301896] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5564.309848] ? do_wait_intr+0xf0/0xf0 [ 5564.314929] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5564.317895] mdc_reint+0x2d/0x110 [mdc] [ 5564.319735] mdc_unlink+0x658/0x870 [mdc] [ 5564.321648] lmv_unlink+0x34d/0xbd0 [lmv] [ 5564.322942] ll_unlink+0x24d/0x7f0 [lustre] [ 5564.324111] vfs_unlink+0x202/0x330 [ 5564.326955] do_unlinkat+0x2fa/0x460 [ 5564.332342] __x64_sys_unlink+0x37/0x50 [ 5564.334164] do_syscall_64+0xc1/0x440 [ 5564.336448] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5564.347602] RIP: 0033:0x7f83e5f5c10b [ 5564.351069] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5564.369406] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5564.379026] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5564.384142] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5564.391909] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5564.398511] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5564.405157] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5572.060005] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5572.062845] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5572.081455] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5572.086387] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5572.090111] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5572.093290] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 5572.095315] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 5572.101565] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5572.111210] RAX: 0000000000004dce RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5572.116802] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5572.120018] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5572.122911] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5572.125923] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5572.130563] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5572.139272] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5572.142603] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5572.146454] Call Trace: [ 5572.154512] [ 5572.155276] ? show_regs.cold.9+0x22/0x2f [ 5572.158667] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5572.161281] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5572.163441] ? watchdog+0x40/0x40 [ 5572.165278] ? hrtimer_interrupt+0x128/0x2c0 [ 5572.167388] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5572.171615] ? apic_timer_interrupt+0xf/0x20 [ 5572.177087] [ 5572.178817] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 5572.183173] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5572.189561] do_raw_spin_lock+0x177/0x190 [ 5572.191911] _raw_spin_lock+0x21/0x30 [ 5572.193688] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5572.198495] process_one_work+0x2c8/0x700 [ 5572.210194] worker_thread+0x296/0x6e0 [ 5572.211612] ? rescuer_thread+0x570/0x570 [ 5572.216946] kthread+0x1d1/0x200 [ 5572.218110] ? set_kthread_struct+0x70/0x70 [ 5572.222449] ret_from_fork+0x1f/0x30 [ 5576.060155] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 5576.063829] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5576.098322] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5576.103406] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5576.107452] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5576.109961] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5576.117701] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5576.120888] RAX: 0000000000007e8d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5576.124330] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5576.130011] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5576.134840] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5576.147136] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5576.151468] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5576.165233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5576.171210] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5576.178306] Call Trace: [ 5576.180945] [ 5576.182311] ? show_regs.cold.9+0x22/0x2f [ 5576.183761] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5576.185243] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5576.187165] ? watchdog+0x40/0x40 [ 5576.188931] ? hrtimer_interrupt+0x128/0x2c0 [ 5576.190897] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5576.192953] ? apic_timer_interrupt+0xf/0x20 [ 5576.195602] [ 5576.196389] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5576.206991] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5576.214850] do_raw_spin_lock+0x177/0x190 [ 5576.222988] _raw_spin_lock+0x21/0x30 [ 5576.228182] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5576.232913] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5576.236872] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5576.243495] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5576.251380] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5576.256970] ? woken_wake_function+0x30/0x30 [ 5576.260336] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5576.270324] kthread+0x1d1/0x200 [ 5576.271675] ? set_kthread_struct+0x70/0x70 [ 5576.278669] ret_from_fork+0x1f/0x30 [ 5584.057634] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 5584.071291] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5584.121669] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5584.136119] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5584.146655] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5584.153799] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5584.172560] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5584.183876] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5584.188570] RDX: 0000000000005463 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5584.197710] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5584.202551] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5584.207846] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5584.216529] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5584.223733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5584.230691] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5584.242201] Call Trace: [ 5584.244838] [ 5584.246883] ? show_regs.cold.9+0x22/0x2f [ 5584.251599] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5584.256726] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5584.261403] ? watchdog+0x40/0x40 [ 5584.265478] ? hrtimer_interrupt+0x128/0x2c0 [ 5584.271854] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5584.278491] ? apic_timer_interrupt+0xf/0x20 [ 5584.285842] [ 5584.287934] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5584.294165] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5584.297203] do_raw_spin_lock+0x177/0x190 [ 5584.301705] _raw_spin_lock+0x21/0x30 [ 5584.304326] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5584.309434] obd_mod_free+0x46/0x140 [ptlrpc] [ 5584.314032] mdc_commit_open+0x91/0xb0 [mdc] [ 5584.318174] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5584.323647] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5584.327625] after_reply+0xe2c/0x1780 [ptlrpc] [ 5584.333509] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5584.337939] ? __next_timer_interrupt+0x160/0x160 [ 5584.342565] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5584.346542] ? do_wait_intr+0xf0/0xf0 [ 5584.350775] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5584.355968] kthread+0x1d1/0x200 [ 5584.358809] ? set_kthread_struct+0x70/0x70 [ 5584.361679] ret_from_fork+0x1f/0x30 [ 5588.462018] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3075s! [ 5588.473225] Showing busy workqueues and worker pools: [ 5588.479121] workqueue events: flags=0x0 [ 5588.480677] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5588.483369] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5588.487322] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5588.494186] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5588.500710] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5588.505983] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5588.516354] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5588.520594] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5588.526482] workqueue events_long: flags=0x0 [ 5588.528717] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5588.532294] pending: lu_objects_flush [obdclass] [ 5588.535731] workqueue events_unbound: flags=0x2 [ 5588.538702] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5588.544587] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5588.552647] workqueue events_power_efficient: flags=0x80 [ 5588.557620] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5588.561247] pending: neigh_periodic_work, do_cache_clean [ 5588.563627] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5588.567176] pending: neigh_periodic_work, check_lifetime [ 5588.571338] workqueue mm_percpu_wq: flags=0x8 [ 5588.575461] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5588.578695] pending: vmstat_update, lru_add_drain_per_cpu [ 5588.581346] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5588.585613] pending: vmstat_update [ 5588.587202] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5588.590533] pending: lru_add_drain_per_cpu BAR(48) [ 5588.592782] workqueue ipv6_addrconf: flags=0x40008 [ 5588.595064] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5588.598168] pending: addrconf_verify_work [ 5588.599913] workqueue knbd0-recv: flags=0x1a [ 5588.601850] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5588.604978] in-flight: 650:recv_work [ 5588.606709] workqueue ptlrpc_pinger: flags=0x2 [ 5588.608340] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5588.610597] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5588.613363] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3075s workers=4 idle: 36 466 29757 [ 5588.616755] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5578s workers=2 idle: 652 [ 5592.062031] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 5592.064808] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5592.081253] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5592.084589] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5592.087049] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5592.088229] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5592.094035] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5592.097468] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5592.101801] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5592.105457] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5592.108884] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5592.112726] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5592.116406] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5592.120744] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5592.123773] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5592.127066] Call Trace: [ 5592.128145] [ 5592.129093] ? show_regs.cold.9+0x22/0x2f [ 5592.130778] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5592.133130] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5592.135261] ? watchdog+0x40/0x40 [ 5592.136792] ? hrtimer_interrupt+0x128/0x2c0 [ 5592.138553] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5592.140818] ? apic_timer_interrupt+0xf/0x20 [ 5592.142666] [ 5592.143487] ? native_safe_halt+0x2e/0x50 [ 5592.145784] kvm_wait+0xcd/0xe0 [ 5592.147810] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5592.151860] do_raw_spin_lock+0x177/0x190 [ 5592.153388] _raw_spin_lock+0x21/0x30 [ 5592.155669] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5592.158264] ? do_raw_spin_unlock+0x75/0x190 [ 5592.160360] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5592.163507] ? do_wait_intr+0xf0/0xf0 [ 5592.165044] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5592.167308] mdc_reint+0x2d/0x110 [mdc] [ 5592.168838] mdc_unlink+0x658/0x870 [mdc] [ 5592.170461] lmv_unlink+0x34d/0xbd0 [lmv] [ 5592.172669] ll_unlink+0x24d/0x7f0 [lustre] [ 5592.174721] vfs_unlink+0x202/0x330 [ 5592.176457] do_unlinkat+0x2fa/0x460 [ 5592.179509] __x64_sys_unlink+0x37/0x50 [ 5592.181464] do_syscall_64+0xc1/0x440 [ 5592.183357] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5592.185734] RIP: 0033:0x7f83e5f5c10b [ 5592.187191] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5592.194655] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5592.198241] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5592.201520] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5592.204856] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5592.208757] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5592.212333] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5600.067189] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5600.072419] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5600.137296] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5600.154638] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5600.157492] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5600.172471] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5600.177325] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5600.202265] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5600.214847] RAX: 0000000000002986 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5600.221925] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5600.234299] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5600.243619] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5600.246266] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5600.260320] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5600.263271] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5600.273923] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5600.285290] Call Trace: [ 5600.286165] [ 5600.286836] ? show_regs.cold.9+0x22/0x2f [ 5600.292653] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5600.296672] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5600.305203] ? watchdog+0x40/0x40 [ 5600.306340] ? hrtimer_interrupt+0x128/0x2c0 [ 5600.325405] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5600.332818] ? apic_timer_interrupt+0xf/0x20 [ 5600.346438] [ 5600.347197] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5600.364183] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5600.366281] do_raw_spin_lock+0x177/0x190 [ 5600.378657] _raw_spin_lock+0x21/0x30 [ 5600.385698] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5600.400662] process_one_work+0x2c8/0x700 [ 5600.402053] worker_thread+0x296/0x6e0 [ 5600.403322] ? rescuer_thread+0x570/0x570 [ 5600.427799] kthread+0x1d1/0x200 [ 5600.431260] ? set_kthread_struct+0x70/0x70 [ 5600.434684] ret_from_fork+0x1f/0x30 [ 5604.062223] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 5604.065067] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5604.087477] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5604.099614] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5604.108688] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5604.115578] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5604.128897] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5604.135767] RAX: 0000000000007e95 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5604.140104] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5604.144009] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5604.147496] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5604.153318] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5604.157214] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5604.162783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5604.169263] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5604.173149] Call Trace: [ 5604.174484] [ 5604.175324] ? show_regs.cold.9+0x22/0x2f [ 5604.176756] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5604.179201] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5604.183158] ? watchdog+0x40/0x40 [ 5604.185675] ? hrtimer_interrupt+0x128/0x2c0 [ 5604.187217] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5604.192805] ? apic_timer_interrupt+0xf/0x20 [ 5604.199211] [ 5604.200244] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5604.207293] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5604.210515] do_raw_spin_lock+0x177/0x190 [ 5604.216736] _raw_spin_lock+0x21/0x30 [ 5604.221805] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5604.231283] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5604.238531] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5604.248658] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5604.254762] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5604.260453] ? woken_wake_function+0x30/0x30 [ 5604.263846] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5604.267463] kthread+0x1d1/0x200 [ 5604.268708] ? set_kthread_struct+0x70/0x70 [ 5604.271174] ret_from_fork+0x1f/0x30 [ 5616.056019] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 5616.058189] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5616.071120] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5616.074639] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5616.077514] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5616.079244] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5616.087159] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5616.091104] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5616.093768] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5616.096006] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5616.098234] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5616.100715] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5616.103227] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5616.106158] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5616.107845] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5616.110207] Call Trace: [ 5616.111100] [ 5616.111734] ? show_regs.cold.9+0x22/0x2f [ 5616.112836] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5616.115203] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5616.116780] ? watchdog+0x40/0x40 [ 5616.118378] ? hrtimer_interrupt+0x128/0x2c0 [ 5616.120630] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5616.121832] ? apic_timer_interrupt+0xf/0x20 [ 5616.124009] [ 5616.125812] ? native_safe_halt+0x2e/0x50 [ 5616.127362] kvm_wait+0xcd/0xe0 [ 5616.128276] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5616.129800] do_raw_spin_lock+0x177/0x190 [ 5616.131255] _raw_spin_lock+0x21/0x30 [ 5616.132916] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5616.134813] obd_mod_free+0x46/0x140 [ptlrpc] [ 5616.138904] mdc_commit_open+0x91/0xb0 [mdc] [ 5616.140245] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5616.142637] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5616.144836] after_reply+0xe2c/0x1780 [ptlrpc] [ 5616.146610] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5616.148241] ? __next_timer_interrupt+0x160/0x160 [ 5616.149617] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5616.151513] ? do_wait_intr+0xf0/0xf0 [ 5616.153236] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5616.155217] kthread+0x1d1/0x200 [ 5616.156650] ? set_kthread_struct+0x70/0x70 [ 5616.158105] ret_from_fork+0x1f/0x30 [ 5618.658336] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3105s! [ 5618.667915] Showing busy workqueues and worker pools: [ 5618.672258] workqueue events: flags=0x0 [ 5618.674826] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5618.680242] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5618.685475] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5618.691162] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5618.699561] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5618.703075] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5618.709020] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5618.718128] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5618.723982] workqueue events_long: flags=0x0 [ 5618.727538] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5618.739909] pending: lu_objects_flush [obdclass] [ 5618.742263] workqueue events_unbound: flags=0x2 [ 5618.745231] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5618.748850] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5618.751424] workqueue events_power_efficient: flags=0x80 [ 5618.755200] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5618.760570] pending: neigh_periodic_work, do_cache_clean [ 5618.763709] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5618.768511] pending: neigh_periodic_work, check_lifetime [ 5618.776194] workqueue mm_percpu_wq: flags=0x8 [ 5618.778840] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5618.781175] pending: vmstat_update, lru_add_drain_per_cpu [ 5618.783009] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5618.786850] pending: vmstat_update [ 5618.790453] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5618.794039] pending: lru_add_drain_per_cpu BAR(48) [ 5618.796462] workqueue ipv6_addrconf: flags=0x40008 [ 5618.801967] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5618.805482] pending: addrconf_verify_work [ 5618.810459] workqueue knbd0-recv: flags=0x1a [ 5618.812847] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5618.816432] in-flight: 650:recv_work [ 5618.818398] workqueue ptlrpc_pinger: flags=0x2 [ 5618.820393] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5618.823213] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5618.825549] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3105s workers=4 idle: 36 466 29757 [ 5618.829344] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5608s workers=2 idle: 652 [ 5620.062855] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 5620.065468] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5620.085890] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5620.091977] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5620.096461] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5620.099541] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5620.108353] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5620.111803] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5620.115702] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5620.119087] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5620.126307] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5620.130981] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5620.137703] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5620.141790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5620.146178] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5620.152529] Call Trace: [ 5620.154019] [ 5620.154976] ? show_regs.cold.9+0x22/0x2f [ 5620.157217] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5620.163118] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5620.167271] ? watchdog+0x40/0x40 [ 5620.170591] ? hrtimer_interrupt+0x128/0x2c0 [ 5620.173610] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5620.175342] ? apic_timer_interrupt+0xf/0x20 [ 5620.182015] [ 5620.185035] ? native_safe_halt+0x2e/0x50 [ 5620.189914] kvm_wait+0xcd/0xe0 [ 5620.191917] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5620.194398] do_raw_spin_lock+0x177/0x190 [ 5620.196287] _raw_spin_lock+0x21/0x30 [ 5620.202084] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5620.205108] ? do_raw_spin_unlock+0x75/0x190 [ 5620.207163] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5620.210148] ? do_wait_intr+0xf0/0xf0 [ 5620.211877] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5620.214180] mdc_reint+0x2d/0x110 [mdc] [ 5620.215996] mdc_unlink+0x658/0x870 [mdc] [ 5620.217942] lmv_unlink+0x34d/0xbd0 [lmv] [ 5620.219562] ll_unlink+0x24d/0x7f0 [lustre] [ 5620.221808] vfs_unlink+0x202/0x330 [ 5620.223338] do_unlinkat+0x2fa/0x460 [ 5620.225801] __x64_sys_unlink+0x37/0x50 [ 5620.228072] do_syscall_64+0xc1/0x440 [ 5620.230226] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5620.232961] RIP: 0033:0x7f83e5f5c10b [ 5620.235604] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5620.252474] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5620.258319] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5620.262863] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5620.266562] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5620.268950] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5620.272092] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5628.060119] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5628.062707] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5628.105998] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5628.120172] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5628.132922] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5628.143676] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5628.150577] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5628.167451] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5628.174216] RAX: 0000000000006edc RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5628.181834] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5628.189075] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5628.197684] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5628.205000] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5628.210794] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5628.216982] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5628.220042] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5628.228746] Call Trace: [ 5628.229793] [ 5628.230648] ? show_regs.cold.9+0x22/0x2f [ 5628.232700] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5628.234899] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5628.238603] ? watchdog+0x40/0x40 [ 5628.239671] ? hrtimer_interrupt+0x128/0x2c0 [ 5628.241005] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5628.253492] ? apic_timer_interrupt+0xf/0x20 [ 5628.257824] [ 5628.259797] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5628.265419] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5628.271543] do_raw_spin_lock+0x177/0x190 [ 5628.274113] _raw_spin_lock+0x21/0x30 [ 5628.277818] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5628.282259] process_one_work+0x2c8/0x700 [ 5628.288169] worker_thread+0x296/0x6e0 [ 5628.292827] ? rescuer_thread+0x570/0x570 [ 5628.294234] kthread+0x1d1/0x200 [ 5628.295410] ? set_kthread_struct+0x70/0x70 [ 5628.298809] ret_from_fork+0x1f/0x30 [ 5632.059271] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 5632.063621] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5632.083626] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5632.087953] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5632.091164] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5632.094044] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5632.100144] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5632.103130] RAX: 0000000000004320 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5632.106230] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5632.110009] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5632.113010] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5632.116185] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5632.119375] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5632.123138] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5632.125193] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5632.127830] Call Trace: [ 5632.128732] [ 5632.129363] ? show_regs.cold.9+0x22/0x2f [ 5632.130709] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5632.132722] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5632.134458] ? watchdog+0x40/0x40 [ 5632.135450] ? hrtimer_interrupt+0x128/0x2c0 [ 5632.137052] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5632.141376] ? apic_timer_interrupt+0xf/0x20 [ 5632.143616] [ 5632.144916] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5632.146850] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5632.149418] do_raw_spin_lock+0x177/0x190 [ 5632.150948] _raw_spin_lock+0x21/0x30 [ 5632.152289] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5632.155334] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5632.158043] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5632.160160] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5632.163155] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5632.165282] ? woken_wake_function+0x30/0x30 [ 5632.166629] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5632.168905] kthread+0x1d1/0x200 [ 5632.170068] ? set_kthread_struct+0x70/0x70 [ 5632.171567] ret_from_fork+0x1f/0x30 [ 5641.042358] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 5641.053729] rcu: 0-....: (2316794 ticks this GP) idle=30a/1/0x4000000000000000 softirq=128519/128519 fqs=126395 [ 5641.065916] (detected by 1, t=3660083 jiffies, g=262441, q=58181) [ 5641.075779] Sending NMI from CPU 1 to CPUs 0: [ 5641.082588] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 5641.085632] rcu: rcu_sched kthread starved for 3129270 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 5641.094375] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5641.100032] rcu: RCU grace-period kthread stack dump: [ 5641.102668] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5641.108532] Call Trace: [ 5641.109661] __schedule+0x351/0xcb0 [ 5641.111394] schedule+0xc0/0x180 [ 5641.113108] schedule_timeout+0xb4/0x190 [ 5641.115117] ? do_raw_spin_unlock+0x75/0x190 [ 5641.117077] ? __next_timer_interrupt+0x160/0x160 [ 5641.119462] ? prepare_to_swait_event+0xf3/0x140 [ 5641.122276] rcu_gp_kthread+0x8dd/0x15b0 [ 5641.124443] ? force_qs_rnp+0x2c0/0x2c0 [ 5641.127588] kthread+0x1d1/0x200 [ 5641.129509] ? set_kthread_struct+0x70/0x70 [ 5641.131888] ret_from_fork+0x1f/0x30 [ 5641.134538] rcu: Stack dump where RCU GP kthread last ran: [ 5641.137350] NMI backtrace for cpu 1 [ 5641.138988] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5641.142902] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5641.145988] Call Trace: [ 5641.146990] [ 5641.147735] ? dump_stack+0xbb/0x10e [ 5641.149211] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5641.151745] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5641.154327] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5641.157135] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5641.160702] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5641.163279] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 5641.166018] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [ 5641.169853] ? update_process_times+0x6d/0xc0 [ 5641.172167] ? tick_sched_handle.isra.18+0x27/0x90 [ 5641.175271] ? tick_sched_timer+0x53/0xb0 [ 5641.177238] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5641.179607] ? tick_sched_handle.isra.18+0x90/0x90 [ 5641.182010] ? hrtimer_interrupt+0x128/0x2c0 [ 5641.183897] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5641.186048] ? apic_timer_interrupt+0xf/0x20 [ 5641.188075] [ 5641.192123] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5641.195024] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5641.200592] ? do_raw_spin_lock+0x177/0x190 [ 5641.204800] ? _raw_spin_lock+0x21/0x30 [ 5641.206521] ? ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5641.210452] ? ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5641.212904] ? ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5641.215672] ? ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5641.218351] ? ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5641.221013] ? woken_wake_function+0x30/0x30 [ 5641.223159] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5641.225961] ? kthread+0x1d1/0x200 [ 5641.227397] ? set_kthread_struct+0x70/0x70 [ 5641.229156] ? ret_from_fork+0x1f/0x30 [ 5648.062022] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 5648.064286] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5648.077859] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5648.080975] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5648.083676] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5648.084942] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5648.090254] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5648.092466] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5648.094800] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5648.096995] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5648.099233] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5648.101260] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5648.103777] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5648.106179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5648.108082] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5648.109888] Call Trace: [ 5648.110639] [ 5648.111403] ? show_regs.cold.9+0x22/0x2f [ 5648.112796] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5648.114215] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5648.115657] ? watchdog+0x40/0x40 [ 5648.116554] ? hrtimer_interrupt+0x128/0x2c0 [ 5648.117814] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5648.119321] ? apic_timer_interrupt+0xf/0x20 [ 5648.120612] [ 5648.121270] ? native_safe_halt+0x2e/0x50 [ 5648.122461] kvm_wait+0xcd/0xe0 [ 5648.123570] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5648.125135] do_raw_spin_lock+0x177/0x190 [ 5648.126330] _raw_spin_lock+0x21/0x30 [ 5648.127484] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5648.129419] ? do_raw_spin_unlock+0x75/0x190 [ 5648.130972] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5648.132548] ? do_wait_intr+0xf0/0xf0 [ 5648.133779] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5648.135495] mdc_reint+0x2d/0x110 [mdc] [ 5648.136657] mdc_unlink+0x658/0x870 [mdc] [ 5648.138080] lmv_unlink+0x34d/0xbd0 [lmv] [ 5648.139259] ll_unlink+0x24d/0x7f0 [lustre] [ 5648.140723] vfs_unlink+0x202/0x330 [ 5648.141927] do_unlinkat+0x2fa/0x460 [ 5648.142887] __x64_sys_unlink+0x37/0x50 [ 5648.144035] do_syscall_64+0xc1/0x440 [ 5648.145178] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5648.146644] RIP: 0033:0x7f83e5f5c10b [ 5648.147684] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5648.153837] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5648.156243] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5648.158488] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5648.160798] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5648.163598] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5648.166055] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5648.874776] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3135s! [ 5648.878221] Showing busy workqueues and worker pools: [ 5648.880306] workqueue events: flags=0x0 [ 5648.881756] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5648.889333] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5648.896846] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5648.899312] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5648.902452] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5648.904710] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5648.908824] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5648.912136] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5648.919982] workqueue events_long: flags=0x0 [ 5648.921420] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5648.930860] pending: lu_objects_flush [obdclass] [ 5648.935189] workqueue events_unbound: flags=0x2 [ 5648.936835] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5648.941334] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5648.944518] workqueue events_power_efficient: flags=0x80 [ 5648.949459] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5648.956811] pending: neigh_periodic_work, do_cache_clean [ 5648.959659] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5648.963856] pending: neigh_periodic_work, check_lifetime [ 5648.970718] workqueue mm_percpu_wq: flags=0x8 [ 5648.973647] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5648.979907] pending: vmstat_update, lru_add_drain_per_cpu [ 5648.982937] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5648.986239] pending: vmstat_update [ 5648.988350] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5648.991547] pending: lru_add_drain_per_cpu BAR(48) [ 5648.993780] workqueue ipv6_addrconf: flags=0x40008 [ 5648.995497] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5649.001811] pending: addrconf_verify_work [ 5649.004421] workqueue knbd0-recv: flags=0x1a [ 5649.005644] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5649.009192] in-flight: 650:recv_work [ 5649.011094] workqueue ptlrpc_pinger: flags=0x2 [ 5649.012859] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5649.014874] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5649.017215] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3135s workers=4 idle: 36 466 29757 [ 5649.020492] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5639s workers=2 idle: 652 [ 5656.060014] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5656.062923] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5656.090867] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5656.095535] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5656.101836] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5656.105591] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5656.110838] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5656.129824] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5656.137200] RAX: 0000000000003e87 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5656.140519] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5656.144387] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5656.152430] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5656.154895] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5656.159873] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5656.164412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5656.167609] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5656.170747] Call Trace: [ 5656.172089] [ 5656.172823] ? show_regs.cold.9+0x22/0x2f [ 5656.174286] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5656.175735] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5656.182430] ? watchdog+0x40/0x40 [ 5656.184011] ? hrtimer_interrupt+0x128/0x2c0 [ 5656.185549] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5656.189402] ? apic_timer_interrupt+0xf/0x20 [ 5656.196270] [ 5656.197473] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5656.200698] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5656.203524] do_raw_spin_lock+0x177/0x190 [ 5656.205939] _raw_spin_lock+0x21/0x30 [ 5656.207734] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5656.211894] process_one_work+0x2c8/0x700 [ 5656.214102] worker_thread+0x296/0x6e0 [ 5656.216567] ? rescuer_thread+0x570/0x570 [ 5656.220191] kthread+0x1d1/0x200 [ 5656.224694] ? set_kthread_struct+0x70/0x70 [ 5656.242755] ret_from_fork+0x1f/0x30 [ 5668.060259] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 5668.069520] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5668.102489] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5668.106739] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5668.110851] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5668.113404] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5668.119838] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5668.123055] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5668.126059] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5668.130057] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5668.132682] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5668.136760] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5668.140578] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5668.144389] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5668.146844] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5668.149794] Call Trace: [ 5668.150840] [ 5668.151695] ? show_regs.cold.9+0x22/0x2f [ 5668.153570] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5668.155981] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5668.157842] ? watchdog+0x40/0x40 [ 5668.159299] ? hrtimer_interrupt+0x128/0x2c0 [ 5668.161109] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5668.163287] ? apic_timer_interrupt+0xf/0x20 [ 5668.165003] [ 5668.165923] ? native_safe_halt+0x2e/0x50 [ 5668.167675] kvm_wait+0xcd/0xe0 [ 5668.169382] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5668.171511] do_raw_spin_lock+0x177/0x190 [ 5668.173277] _raw_spin_lock+0x21/0x30 [ 5668.174768] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5668.177629] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5668.179981] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5668.182559] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5668.185315] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5668.187927] ? woken_wake_function+0x30/0x30 [ 5668.189707] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5668.192539] kthread+0x1d1/0x200 [ 5668.193914] ? set_kthread_struct+0x70/0x70 [ 5668.195698] ret_from_fork+0x1f/0x30 [ 5676.057321] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 5676.062020] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 5676.063063] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) [ 5676.065243] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) [ 5676.074963] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5676.075016] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5676.075022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5676.075025] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5676.075037] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5676.075041] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5676.075045] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5676.075047] RDX: 0000000000000039 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5676.075049] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5676.075051] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5676.075054] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5676.075057] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5676.075059] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5676.075061] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5676.075068] Call Trace: [ 5676.075073] [ 5676.075078] ? show_regs.cold.9+0x22/0x2f [ 5676.075086] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5676.075143] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5676.075147] ? watchdog+0x40/0x40 [ 5676.075150] ? hrtimer_interrupt+0x128/0x2c0 [ 5676.075154] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5676.075161] ? apic_timer_interrupt+0xf/0x20 [ 5676.075165] [ 5676.075167] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5676.075228] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5676.075233] do_raw_spin_lock+0x177/0x190 [ 5676.075237] _raw_spin_lock+0x21/0x30 [ 5676.075243] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5676.075570] obd_mod_free+0x46/0x140 [ptlrpc] [ 5676.080256] lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5676.080322] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5676.080330] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5676.080333] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5676.080347] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5676.080352] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5676.080356] RAX: 000000000000622c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5676.080359] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5676.080361] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5676.080363] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5676.080366] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5676.080369] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5676.080372] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5676.080374] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5676.080388] Call Trace: [ 5676.080394] [ 5676.080400] ? show_regs.cold.9+0x22/0x2f [ 5676.080409] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5676.080416] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5676.080420] ? watchdog+0x40/0x40 [ 5676.080425] ? hrtimer_interrupt+0x128/0x2c0 [ 5676.080429] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5676.080501] ? apic_timer_interrupt+0xf/0x20 [ 5676.080506] [ 5676.080509] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5676.080513] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5676.080517] do_raw_spin_lock+0x177/0x190 [ 5676.080522] _raw_spin_lock+0x21/0x30 [ 5676.080528] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5676.110667] mdc_commit_open+0x91/0xb0 [mdc] [ 5676.110733] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5676.121468] ? do_raw_spin_unlock+0x75/0x190 [ 5676.121489] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5676.123036] ? do_wait_intr+0xf0/0xf0 [ 5676.123049] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5676.123364] mdc_reint+0x2d/0x110 [mdc] [ 5676.123413] mdc_unlink+0x658/0x870 [mdc] [ 5676.123440] lmv_unlink+0x34d/0xbd0 [lmv] [ 5676.123626] ll_unlink+0x24d/0x7f0 [lustre] [ 5676.123906] vfs_unlink+0x202/0x330 [ 5676.123915] do_unlinkat+0x2fa/0x460 [ 5676.123921] __x64_sys_unlink+0x37/0x50 [ 5676.123924] do_syscall_64+0xc1/0x440 [ 5676.123931] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5676.123940] RIP: 0033:0x7f83e5f5c10b [ 5676.123947] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5676.123951] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5676.123955] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5676.123957] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5676.123959] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5676.123961] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5676.123963] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5676.535218] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5676.539344] after_reply+0xe2c/0x1780 [ptlrpc] [ 5676.541837] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5676.546591] ? __next_timer_interrupt+0x160/0x160 [ 5676.548920] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5676.552704] ? do_wait_intr+0xf0/0xf0 [ 5676.554125] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5676.556921] kthread+0x1d1/0x200 [ 5676.558192] ? set_kthread_struct+0x70/0x70 [ 5676.560444] ret_from_fork+0x1f/0x30 [ 5679.074462] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3165s! [ 5679.085779] Showing busy workqueues and worker pools: [ 5679.094749] workqueue events: flags=0x0 [ 5679.100436] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5679.107550] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5679.122349] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5679.132057] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5679.143519] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5679.153497] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5679.159820] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5679.163541] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5679.168490] workqueue events_long: flags=0x0 [ 5679.170837] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5679.177490] pending: lu_objects_flush [obdclass] [ 5679.185452] workqueue events_unbound: flags=0x2 [ 5679.189452] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5679.196934] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5679.203488] workqueue events_power_efficient: flags=0x80 [ 5679.209809] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5679.215311] pending: neigh_periodic_work, do_cache_clean [ 5679.217734] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5679.221199] pending: neigh_periodic_work, check_lifetime [ 5679.224177] workqueue mm_percpu_wq: flags=0x8 [ 5679.226500] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5679.232233] pending: vmstat_update, lru_add_drain_per_cpu [ 5679.235855] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5679.239835] pending: vmstat_update [ 5679.242063] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5679.246234] pending: lru_add_drain_per_cpu BAR(48) [ 5679.249231] workqueue ipv6_addrconf: flags=0x40008 [ 5679.252166] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5679.260134] pending: addrconf_verify_work [ 5679.264026] workqueue knbd0-recv: flags=0x1a [ 5679.268062] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5679.273049] in-flight: 650:recv_work [ 5679.275011] workqueue ptlrpc_pinger: flags=0x2 [ 5679.277339] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5679.279493] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5679.281515] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3166s workers=4 idle: 36 466 29757 [ 5679.284153] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5669s workers=2 idle: 652 [ 5684.061959] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5684.069396] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5684.128555] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5684.148933] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5684.160272] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5684.167592] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5684.172552] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5684.190847] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5684.199728] RAX: 0000000000006885 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5684.211540] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5684.220692] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5684.236799] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5684.252753] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5684.260987] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5684.271511] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5684.278576] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5684.298407] Call Trace: [ 5684.299139] [ 5684.299878] ? show_regs.cold.9+0x22/0x2f [ 5684.311761] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5684.318344] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5684.319771] ? watchdog+0x40/0x40 [ 5684.332201] ? hrtimer_interrupt+0x128/0x2c0 [ 5684.336928] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5684.343479] ? apic_timer_interrupt+0xf/0x20 [ 5684.347214] [ 5684.348546] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5684.360173] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5684.365915] do_raw_spin_lock+0x177/0x190 [ 5684.373447] _raw_spin_lock+0x21/0x30 [ 5684.380928] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5684.384491] process_one_work+0x2c8/0x700 [ 5684.385832] worker_thread+0x296/0x6e0 [ 5684.386889] ? rescuer_thread+0x570/0x570 [ 5684.393761] kthread+0x1d1/0x200 [ 5684.396370] ? set_kthread_struct+0x70/0x70 [ 5684.398498] ret_from_fork+0x1f/0x30 [ 5696.061224] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 5696.068951] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5696.103850] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5696.109328] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5696.113289] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5696.116168] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5696.126172] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5696.130963] RAX: 0000000000005296 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5696.140775] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5696.146384] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5696.151844] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5696.157771] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5696.162308] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5696.171772] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5696.175997] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5696.182375] Call Trace: [ 5696.184002] [ 5696.186759] ? show_regs.cold.9+0x22/0x2f [ 5696.189756] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5696.194732] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5696.200648] ? watchdog+0x40/0x40 [ 5696.203722] ? hrtimer_interrupt+0x128/0x2c0 [ 5696.205798] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5696.212333] ? apic_timer_interrupt+0xf/0x20 [ 5696.216414] [ 5696.218682] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5696.233954] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5696.236846] do_raw_spin_lock+0x177/0x190 [ 5696.239671] _raw_spin_lock+0x21/0x30 [ 5696.242795] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5696.247539] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5696.250313] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5696.253300] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5696.256004] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5696.258809] ? woken_wake_function+0x30/0x30 [ 5696.260656] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5696.263474] kthread+0x1d1/0x200 [ 5696.264586] ? set_kthread_struct+0x70/0x70 [ 5696.265786] ret_from_fork+0x1f/0x30 [ 5704.056223] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 5704.060030] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5704.066185] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 5704.088344] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5704.088360] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5704.088363] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5704.088406] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5704.088409] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5704.088415] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5704.088486] RDX: 000000000000640e RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5704.088488] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5704.088491] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5704.088493] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5704.088497] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5704.088500] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5704.088502] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5704.088510] Call Trace: [ 5704.088527] [ 5704.088533] ? show_regs.cold.9+0x22/0x2f [ 5704.088544] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5704.088550] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5704.088555] ? watchdog+0x40/0x40 [ 5704.088559] ? hrtimer_interrupt+0x128/0x2c0 [ 5704.088563] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5704.088570] ? apic_timer_interrupt+0xf/0x20 [ 5704.088675] [ 5704.088678] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5704.088682] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5704.088687] do_raw_spin_lock+0x177/0x190 [ 5704.088691] _raw_spin_lock+0x21/0x30 [ 5704.088699] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5704.089307] obd_mod_free+0x46/0x140 [ptlrpc] [ 5704.089507] mdc_commit_open+0x91/0xb0 [mdc] [ 5704.089551] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5704.093250] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5704.093332] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5704.093339] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5704.093343] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5704.093357] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5704.098593] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5704.101755] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5704.101768] RAX: 0000000000007b96 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5704.104665] after_reply+0xe2c/0x1780 [ptlrpc] [ 5704.111549] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5704.111557] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5704.111560] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5704.111563] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5704.111566] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5704.111570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5704.111572] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5704.111580] Call Trace: [ 5704.115056] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5704.115398] ? __next_timer_interrupt+0x160/0x160 [ 5704.115408] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5704.115638] ? do_wait_intr+0xf0/0xf0 [ 5704.115647] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5704.115858] kthread+0x1d1/0x200 [ 5704.115923] ? set_kthread_struct+0x70/0x70 [ 5704.115927] ret_from_fork+0x1f/0x30 [ 5704.314602] [ 5704.315794] ? show_regs.cold.9+0x22/0x2f [ 5704.317831] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5704.320108] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5704.322201] ? watchdog+0x40/0x40 [ 5704.324159] ? hrtimer_interrupt+0x128/0x2c0 [ 5704.326322] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5704.328657] ? apic_timer_interrupt+0xf/0x20 [ 5704.331123] [ 5704.332245] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5704.334609] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5704.337304] do_raw_spin_lock+0x177/0x190 [ 5704.339356] _raw_spin_lock+0x21/0x30 [ 5704.341238] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5704.343883] ? do_raw_spin_unlock+0x75/0x190 [ 5704.347765] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5704.352731] ? do_wait_intr+0xf0/0xf0 [ 5704.355271] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5704.360159] mdc_reint+0x2d/0x110 [mdc] [ 5704.361604] mdc_unlink+0x658/0x870 [mdc] [ 5704.363193] lmv_unlink+0x34d/0xbd0 [lmv] [ 5704.365135] ll_unlink+0x24d/0x7f0 [lustre] [ 5704.367416] vfs_unlink+0x202/0x330 [ 5704.369358] do_unlinkat+0x2fa/0x460 [ 5704.371289] __x64_sys_unlink+0x37/0x50 [ 5704.373386] do_syscall_64+0xc1/0x440 [ 5704.375296] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5704.378246] RIP: 0033:0x7f83e5f5c10b [ 5704.379876] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5704.404804] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5704.413803] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5704.419377] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5704.428011] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5704.434323] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5704.441801] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5709.792878] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3196s! [ 5709.799336] Showing busy workqueues and worker pools: [ 5709.807850] workqueue events: flags=0x0 [ 5709.812807] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5709.818646] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5709.837416] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5709.852447] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5709.864230] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5709.870260] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5709.876324] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5709.881522] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5709.890337] workqueue events_long: flags=0x0 [ 5709.893371] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5709.898286] pending: lu_objects_flush [obdclass] [ 5709.901128] workqueue events_unbound: flags=0x2 [ 5709.903307] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5709.908432] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5709.914512] workqueue events_power_efficient: flags=0x80 [ 5709.923250] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5709.929307] pending: neigh_periodic_work, do_cache_clean [ 5709.940766] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5709.944281] pending: neigh_periodic_work, check_lifetime [ 5709.946390] workqueue mm_percpu_wq: flags=0x8 [ 5709.951072] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5709.957304] pending: vmstat_update, lru_add_drain_per_cpu [ 5709.963019] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5709.973442] pending: vmstat_update [ 5709.975254] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5709.981554] pending: lru_add_drain_per_cpu BAR(48) [ 5709.986202] workqueue ipv6_addrconf: flags=0x40008 [ 5709.993064] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5709.999548] pending: addrconf_verify_work [ 5710.001106] workqueue knbd0-recv: flags=0x1a [ 5710.007502] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5710.013063] in-flight: 650:recv_work [ 5710.018264] workqueue ptlrpc_pinger: flags=0x2 [ 5710.021909] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5710.025332] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5710.028235] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3196s workers=4 idle: 36 466 29757 [ 5710.034055] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5700s workers=2 idle: 652 [ 5712.062571] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5712.073196] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5712.111191] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5712.123154] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5712.131097] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5712.137655] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5712.140971] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5712.150204] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5712.159393] RAX: 0000000000006f1f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5712.165949] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5712.170899] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5712.173368] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5712.179668] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5712.186225] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5712.194147] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5712.197723] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5712.204961] Call Trace: [ 5712.208077] [ 5712.211245] ? show_regs.cold.9+0x22/0x2f [ 5712.216042] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5712.219608] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5712.222996] ? watchdog+0x40/0x40 [ 5712.225167] ? hrtimer_interrupt+0x128/0x2c0 [ 5712.227295] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5712.232055] ? apic_timer_interrupt+0xf/0x20 [ 5712.234448] [ 5712.237382] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5712.242343] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5712.246001] do_raw_spin_lock+0x177/0x190 [ 5712.250050] _raw_spin_lock+0x21/0x30 [ 5712.253325] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5712.255305] process_one_work+0x2c8/0x700 [ 5712.258072] worker_thread+0x296/0x6e0 [ 5712.261030] ? rescuer_thread+0x570/0x570 [ 5712.264046] kthread+0x1d1/0x200 [ 5712.267301] ? set_kthread_struct+0x70/0x70 [ 5712.269048] ret_from_fork+0x1f/0x30 [ 5724.060684] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 5724.071543] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5724.117442] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5724.131108] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5724.136439] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5724.139481] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5724.165878] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5724.170419] RAX: 0000000000005f8c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5724.179132] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5724.184012] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5724.187972] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5724.192419] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5724.197620] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5724.205238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5724.214038] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5724.230295] Call Trace: [ 5724.232311] [ 5724.234485] ? show_regs.cold.9+0x22/0x2f [ 5724.249600] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5724.254546] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5724.257295] ? watchdog+0x40/0x40 [ 5724.260525] ? hrtimer_interrupt+0x128/0x2c0 [ 5724.265944] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5724.277548] ? apic_timer_interrupt+0xf/0x20 [ 5724.284349] [ 5724.289286] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5724.294013] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5724.297459] do_raw_spin_lock+0x177/0x190 [ 5724.303547] _raw_spin_lock+0x21/0x30 [ 5724.308112] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5724.315138] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5724.318259] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5724.321432] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5724.330920] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5724.339606] ? woken_wake_function+0x30/0x30 [ 5724.344059] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5724.350833] kthread+0x1d1/0x200 [ 5724.353308] ? set_kthread_struct+0x70/0x70 [ 5724.361636] ret_from_fork+0x1f/0x30 [ 5732.062020] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5732.065879] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5732.098023] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5732.107304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5732.120723] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5732.125881] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5732.137025] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5732.142295] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5732.148379] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5732.156225] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5732.166232] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5732.171297] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5732.184257] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5732.190027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5732.195303] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5732.205256] Call Trace: [ 5732.209333] [ 5732.214182] ? show_regs.cold.9+0x22/0x2f [ 5732.216700] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5732.221666] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5732.223516] ? watchdog+0x40/0x40 [ 5732.224775] ? hrtimer_interrupt+0x128/0x2c0 [ 5732.227379] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5732.229784] ? apic_timer_interrupt+0xf/0x20 [ 5732.231470] [ 5732.232334] ? native_safe_halt+0x2e/0x50 [ 5732.235257] kvm_wait+0xcd/0xe0 [ 5732.236520] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5732.243754] do_raw_spin_lock+0x177/0x190 [ 5732.249393] _raw_spin_lock+0x21/0x30 [ 5732.251008] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5732.254604] ? do_raw_spin_unlock+0x75/0x190 [ 5732.257270] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5732.259479] ? do_wait_intr+0xf0/0xf0 [ 5732.264643] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5732.267060] mdc_reint+0x2d/0x110 [mdc] [ 5732.271988] mdc_unlink+0x658/0x870 [mdc] [ 5732.280485] lmv_unlink+0x34d/0xbd0 [lmv] [ 5732.283270] ll_unlink+0x24d/0x7f0 [lustre] [ 5732.289188] vfs_unlink+0x202/0x330 [ 5732.291411] do_unlinkat+0x2fa/0x460 [ 5732.293148] __x64_sys_unlink+0x37/0x50 [ 5732.294656] do_syscall_64+0xc1/0x440 [ 5732.299743] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5732.304954] RIP: 0033:0x7f83e5f5c10b [ 5732.306415] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5732.316709] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5732.324370] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5732.331467] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5732.338083] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5732.343661] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5732.351135] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5736.056618] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 5736.071952] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5736.131166] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5736.149810] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5736.158504] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5736.168684] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5736.178919] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5736.182830] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5736.188136] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5736.194067] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5736.199666] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5736.210200] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5736.218554] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5736.228037] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5736.236683] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5736.246027] Call Trace: [ 5736.247043] [ 5736.248046] ? show_regs.cold.9+0x22/0x2f [ 5736.251560] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5736.256791] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5736.263282] ? watchdog+0x40/0x40 [ 5736.267475] ? hrtimer_interrupt+0x128/0x2c0 [ 5736.269894] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5736.276398] ? apic_timer_interrupt+0xf/0x20 [ 5736.280885] [ 5736.282092] ? native_safe_halt+0x2e/0x50 [ 5736.284851] kvm_wait+0xcd/0xe0 [ 5736.287213] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5736.291694] do_raw_spin_lock+0x177/0x190 [ 5736.295213] _raw_spin_lock+0x21/0x30 [ 5736.299989] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5736.307115] obd_mod_free+0x46/0x140 [ptlrpc] [ 5736.310213] mdc_commit_open+0x91/0xb0 [mdc] [ 5736.312914] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5736.318947] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5736.322920] after_reply+0xe2c/0x1780 [ptlrpc] [ 5736.328842] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5736.334914] ? __next_timer_interrupt+0x160/0x160 [ 5736.337391] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5736.340921] ? do_wait_intr+0xf0/0xf0 [ 5736.343128] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5736.348960] kthread+0x1d1/0x200 [ 5736.350067] ? set_kthread_struct+0x70/0x70 [ 5736.354187] ret_from_fork+0x1f/0x30 [ 5740.060369] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5740.064993] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5740.108993] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5740.128685] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5740.137056] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5740.139572] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5740.143876] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5740.169195] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5740.177689] RAX: 000000000000197f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5740.188214] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5740.201165] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5740.203631] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5740.214171] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5740.223356] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5740.233908] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5740.237362] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5740.243902] Call Trace: [ 5740.248240] [ 5740.249902] ? show_regs.cold.9+0x22/0x2f [ 5740.255396] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5740.257182] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5740.261158] ? watchdog+0x40/0x40 [ 5740.264754] ? hrtimer_interrupt+0x128/0x2c0 [ 5740.269926] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5740.277006] ? apic_timer_interrupt+0xf/0x20 [ 5740.280737] [ 5740.281444] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5740.290144] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5740.292125] do_raw_spin_lock+0x177/0x190 [ 5740.302816] _raw_spin_lock+0x21/0x30 [ 5740.304218] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5740.311598] process_one_work+0x2c8/0x700 [ 5740.321434] worker_thread+0x296/0x6e0 [ 5740.326082] ? rescuer_thread+0x570/0x570 [ 5740.333071] kthread+0x1d1/0x200 [ 5740.338385] ? set_kthread_struct+0x70/0x70 [ 5740.342252] ret_from_fork+0x1f/0x30 [ 5740.513215] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3227s! [ 5740.517029] Showing busy workqueues and worker pools: [ 5740.519593] workqueue events: flags=0x0 [ 5740.521435] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5740.524873] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5740.529812] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5740.533283] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5740.536861] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5740.540567] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5740.544883] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5740.548214] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5740.555049] workqueue events_long: flags=0x0 [ 5740.557982] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5740.561956] pending: lu_objects_flush [obdclass] [ 5740.565860] workqueue events_unbound: flags=0x2 [ 5740.568448] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5740.571887] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5740.575273] workqueue events_power_efficient: flags=0x80 [ 5740.578380] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5740.584741] pending: neigh_periodic_work, do_cache_clean [ 5740.588874] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5740.594614] pending: neigh_periodic_work, check_lifetime [ 5740.598387] workqueue mm_percpu_wq: flags=0x8 [ 5740.603050] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5740.607921] pending: vmstat_update, lru_add_drain_per_cpu [ 5740.611335] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5740.614762] pending: vmstat_update [ 5740.619060] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5740.623637] pending: lru_add_drain_per_cpu BAR(48) [ 5740.626630] workqueue ipv6_addrconf: flags=0x40008 [ 5740.629144] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5740.635961] pending: addrconf_verify_work [ 5740.638991] workqueue knbd0-recv: flags=0x1a [ 5740.641433] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5740.648297] in-flight: 650:recv_work [ 5740.653069] workqueue ptlrpc_pinger: flags=0x2 [ 5740.655441] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5740.660453] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5740.666388] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3227s workers=4 idle: 36 466 29757 [ 5740.673620] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5730s workers=2 idle: 652 [ 5752.059031] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 5752.068928] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5752.116813] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5752.123754] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5752.128545] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5752.134292] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5752.157833] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5752.164477] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5752.170808] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5752.174594] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5752.185209] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5752.190788] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5752.194548] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5752.203132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5752.207019] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5752.210928] Call Trace: [ 5752.212508] [ 5752.218280] ? show_regs.cold.9+0x22/0x2f [ 5752.220630] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5752.228907] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5752.233668] ? watchdog+0x40/0x40 [ 5752.239826] ? hrtimer_interrupt+0x128/0x2c0 [ 5752.245274] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5752.248676] ? apic_timer_interrupt+0xf/0x20 [ 5752.256205] [ 5752.257341] ? native_safe_halt+0x2e/0x50 [ 5752.262715] kvm_wait+0xcd/0xe0 [ 5752.266592] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5752.272976] do_raw_spin_lock+0x177/0x190 [ 5752.275095] _raw_spin_lock+0x21/0x30 [ 5752.278788] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5752.287745] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5752.295146] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5752.300062] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5752.306185] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5752.309201] ? woken_wake_function+0x30/0x30 [ 5752.312939] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5752.317132] kthread+0x1d1/0x200 [ 5752.319812] ? set_kthread_struct+0x70/0x70 [ 5752.321886] ret_from_fork+0x1f/0x30 [ 5760.062013] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5760.065163] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5760.083443] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5760.088572] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5760.093241] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5760.096065] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5760.103849] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5760.107585] RAX: 00000000000003e1 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5760.112761] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5760.116895] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5760.121182] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5760.126029] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5760.130488] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5760.135192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5760.138604] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5760.145811] Call Trace: [ 5760.149624] [ 5760.152960] ? show_regs.cold.9+0x22/0x2f [ 5760.154554] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5760.166756] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5760.174739] ? watchdog+0x40/0x40 [ 5760.176573] ? hrtimer_interrupt+0x128/0x2c0 [ 5760.184717] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5760.189788] ? apic_timer_interrupt+0xf/0x20 [ 5760.194594] [ 5760.195377] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5760.202277] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5760.212837] do_raw_spin_lock+0x177/0x190 [ 5760.214209] _raw_spin_lock+0x21/0x30 [ 5760.215343] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5760.218496] ? do_raw_spin_unlock+0x75/0x190 [ 5760.219930] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5760.223346] ? do_wait_intr+0xf0/0xf0 [ 5760.226815] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5760.231434] mdc_reint+0x2d/0x110 [mdc] [ 5760.234968] mdc_unlink+0x658/0x870 [mdc] [ 5760.241324] lmv_unlink+0x34d/0xbd0 [lmv] [ 5760.244968] ll_unlink+0x24d/0x7f0 [lustre] [ 5760.249855] vfs_unlink+0x202/0x330 [ 5760.253036] do_unlinkat+0x2fa/0x460 [ 5760.256823] __x64_sys_unlink+0x37/0x50 [ 5760.259029] do_syscall_64+0xc1/0x440 [ 5760.260898] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5760.263448] RIP: 0033:0x7f83e5f5c10b [ 5760.264894] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5760.273587] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5760.280716] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5760.292409] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5760.296502] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5760.305571] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5760.309620] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5768.056021] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 5768.060031] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5768.063108] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5768.063177] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5768.063184] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5768.063198] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5768.070752] Modules linked in: lustre(O) [ 5768.118511] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5768.118524] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5768.118530] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5768.118533] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5768.118535] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5768.118538] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5768.118540] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5768.118543] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5768.118547] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5768.118549] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5768.118557] Call Trace: [ 5768.118562] [ 5768.118568] ? show_regs.cold.9+0x22/0x2f [ 5768.128188] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5768.128266] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5768.128273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5768.128279] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5768.133612] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5768.133631] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5768.133637] ? watchdog+0x40/0x40 [ 5768.133641] ? hrtimer_interrupt+0x128/0x2c0 [ 5768.133645] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5768.133655] ? apic_timer_interrupt+0xf/0x20 [ 5768.133660] [ 5768.133665] ? native_safe_halt+0x2e/0x50 [ 5768.133669] kvm_wait+0xcd/0xe0 [ 5768.133678] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5768.133685] do_raw_spin_lock+0x177/0x190 [ 5768.133689] _raw_spin_lock+0x21/0x30 [ 5768.133696] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5768.135621] obd_mod_free+0x46/0x140 [ptlrpc] [ 5768.135838] mdc_commit_open+0x91/0xb0 [mdc] [ 5768.135938] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5768.136156] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5768.136420] after_reply+0xe2c/0x1780 [ptlrpc] [ 5768.136619] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5768.136824] ? __next_timer_interrupt+0x160/0x160 [ 5768.136834] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5768.137607] ? do_wait_intr+0xf0/0xf0 [ 5768.137618] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5768.137830] kthread+0x1d1/0x200 [ 5768.137836] ? set_kthread_struct+0x70/0x70 [ 5768.137840] ret_from_fork+0x1f/0x30 [ 5768.358897] [ 5768.359485] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5768.360518] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5768.364953] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5768.369207] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5768.371546] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5768.379524] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5768.383947] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5768.387550] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5768.390506] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5768.394611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5768.398450] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5768.404374] Call Trace: [ 5768.406370] [ 5768.409864] ? show_regs.cold.9+0x22/0x2f [ 5768.414453] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5768.417810] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5768.420790] ? watchdog+0x40/0x40 [ 5768.422653] ? hrtimer_interrupt+0x128/0x2c0 [ 5768.424157] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5768.426551] ? apic_timer_interrupt+0xf/0x20 [ 5768.430609] [ 5768.432599] ? native_safe_halt+0x2e/0x50 [ 5768.436595] kvm_wait+0xcd/0xe0 [ 5768.439584] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5768.448847] do_raw_spin_lock+0x177/0x190 [ 5768.452608] _raw_spin_lock+0x21/0x30 [ 5768.454808] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5768.459143] process_one_work+0x2c8/0x700 [ 5768.465763] worker_thread+0x296/0x6e0 [ 5768.468561] ? rescuer_thread+0x570/0x570 [ 5768.472680] kthread+0x1d1/0x200 [ 5768.473896] ? set_kthread_struct+0x70/0x70 [ 5768.477160] ret_from_fork+0x1f/0x30 [ 5770.721189] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3257s! [ 5770.725951] Showing busy workqueues and worker pools: [ 5770.728896] workqueue events: flags=0x0 [ 5770.730672] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5770.732858] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5770.736158] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5770.738559] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5770.741371] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5770.744416] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5770.747758] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5770.750623] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5770.753854] workqueue events_long: flags=0x0 [ 5770.755353] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5770.757506] pending: lu_objects_flush [obdclass] [ 5770.759424] workqueue events_unbound: flags=0x2 [ 5770.760688] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5770.763488] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5770.765900] workqueue events_power_efficient: flags=0x80 [ 5770.767805] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5770.770163] pending: neigh_periodic_work, do_cache_clean [ 5770.772825] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5770.775592] pending: neigh_periodic_work, check_lifetime [ 5770.777156] workqueue mm_percpu_wq: flags=0x8 [ 5770.778991] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5770.781417] pending: vmstat_update, lru_add_drain_per_cpu [ 5770.783853] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5770.786356] pending: vmstat_update [ 5770.787659] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5770.790297] pending: lru_add_drain_per_cpu BAR(48) [ 5770.791837] workqueue ipv6_addrconf: flags=0x40008 [ 5770.793445] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5770.795586] pending: addrconf_verify_work [ 5770.797058] workqueue knbd0-recv: flags=0x1a [ 5770.798569] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5770.800695] in-flight: 650:recv_work [ 5770.802360] workqueue ptlrpc_pinger: flags=0x2 [ 5770.803926] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5770.806592] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5770.809654] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3257s workers=4 idle: 36 466 29757 [ 5770.813443] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5760s workers=2 idle: 652 [ 5780.059025] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 5780.065334] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5780.101939] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5780.108471] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5780.116007] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5780.118998] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5780.131054] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5780.134530] RAX: 000000000000722e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5780.136547] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5780.139598] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5780.142005] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5780.145636] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5780.148963] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5780.153942] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5780.158276] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5780.165363] Call Trace: [ 5780.166446] [ 5780.168291] ? show_regs.cold.9+0x22/0x2f [ 5780.170378] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5780.173094] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5780.176811] ? watchdog+0x40/0x40 [ 5780.179833] ? hrtimer_interrupt+0x128/0x2c0 [ 5780.182726] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5780.185723] ? apic_timer_interrupt+0xf/0x20 [ 5780.189824] [ 5780.190453] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5780.193323] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5780.195896] do_raw_spin_lock+0x177/0x190 [ 5780.199578] _raw_spin_lock+0x21/0x30 [ 5780.201956] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5780.207417] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5780.209094] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5780.210834] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5780.212596] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5780.214619] ? woken_wake_function+0x30/0x30 [ 5780.216235] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5780.218631] kthread+0x1d1/0x200 [ 5780.219716] ? set_kthread_struct+0x70/0x70 [ 5780.221442] ret_from_fork+0x1f/0x30 [ 5788.062016] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5788.067220] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5788.103426] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5788.116382] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5788.125891] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5788.134446] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5788.147909] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5788.156257] RAX: 0000000000000c2f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5788.163753] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5788.168448] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5788.174401] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5788.179010] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5788.185666] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5788.188914] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5788.196345] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5788.206500] Call Trace: [ 5788.212274] [ 5788.213490] ? show_regs.cold.9+0x22/0x2f [ 5788.220388] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5788.225495] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5788.230163] ? watchdog+0x40/0x40 [ 5788.232634] ? hrtimer_interrupt+0x128/0x2c0 [ 5788.234946] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5788.237386] ? apic_timer_interrupt+0xf/0x20 [ 5788.238745] [ 5788.239516] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5788.241755] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5788.243987] do_raw_spin_lock+0x177/0x190 [ 5788.245791] _raw_spin_lock+0x21/0x30 [ 5788.247668] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5788.250554] ? do_raw_spin_unlock+0x75/0x190 [ 5788.252900] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5788.255743] ? do_wait_intr+0xf0/0xf0 [ 5788.257561] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5788.260583] mdc_reint+0x2d/0x110 [mdc] [ 5788.262505] mdc_unlink+0x658/0x870 [mdc] [ 5788.264295] lmv_unlink+0x34d/0xbd0 [lmv] [ 5788.266290] ll_unlink+0x24d/0x7f0 [lustre] [ 5788.268143] vfs_unlink+0x202/0x330 [ 5788.269920] do_unlinkat+0x2fa/0x460 [ 5788.271995] __x64_sys_unlink+0x37/0x50 [ 5788.273781] do_syscall_64+0xc1/0x440 [ 5788.275339] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5788.277991] RIP: 0033:0x7f83e5f5c10b [ 5788.279400] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5788.287379] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5788.291489] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5788.295036] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5788.300766] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5788.305393] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5788.309718] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5796.056026] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 5796.059975] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5796.060165] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 5796.080778] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5796.088886] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) [ 5796.099621] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5796.099632] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5796.099647] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5796.099651] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5796.099656] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5796.099659] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5796.099661] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5796.099663] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5796.099665] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5796.099668] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5796.099672] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5796.099674] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5796.099682] Call Trace: [ 5796.099687] [ 5796.099693] ? show_regs.cold.9+0x22/0x2f [ 5796.099706] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5796.099712] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5796.099717] ? watchdog+0x40/0x40 [ 5796.099721] ? hrtimer_interrupt+0x128/0x2c0 [ 5796.099739] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5796.099745] ? apic_timer_interrupt+0xf/0x20 [ 5796.099749] [ 5796.099752] ? native_safe_halt+0x2e/0x50 [ 5796.099756] kvm_wait+0xcd/0xe0 [ 5796.099763] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5796.099771] do_raw_spin_lock+0x177/0x190 [ 5796.099776] _raw_spin_lock+0x21/0x30 [ 5796.099782] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5796.105294] obd_mod_free+0x46/0x140 [ptlrpc] [ 5796.105527] mdc_commit_open+0x91/0xb0 [mdc] [ 5796.105563] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5796.105767] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5796.106130] after_reply+0xe2c/0x1780 [ptlrpc] [ 5796.106328] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5796.106718] ? __next_timer_interrupt+0x160/0x160 [ 5796.106737] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5796.107285] ? do_wait_intr+0xf0/0xf0 [ 5796.107295] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5796.123245] fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5796.123302] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5796.123311] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5796.123317] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5796.123839] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5796.123851] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5796.123855] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5796.123860] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5796.123862] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5796.123864] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5796.123867] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5796.123869] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5796.123872] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5796.123876] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5796.123878] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5796.123887] Call Trace: [ 5796.123893] [ 5796.123898] ? show_regs.cold.9+0x22/0x2f [ 5796.123912] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5796.123920] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5796.123934] ? watchdog+0x40/0x40 [ 5796.123940] ? hrtimer_interrupt+0x128/0x2c0 [ 5796.123945] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5796.123952] ? apic_timer_interrupt+0xf/0x20 [ 5796.123957] [ 5796.123960] ? native_safe_halt+0x2e/0x50 [ 5796.123964] kvm_wait+0xcd/0xe0 [ 5796.123973] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5796.123981] do_raw_spin_lock+0x177/0x190 [ 5796.123985] _raw_spin_lock+0x21/0x30 [ 5796.123989] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5796.124306] process_one_work+0x2c8/0x700 [ 5796.124314] worker_thread+0x296/0x6e0 [ 5796.124318] ? rescuer_thread+0x570/0x570 [ 5796.124321] kthread+0x1d1/0x200 [ 5796.124325] ? set_kthread_struct+0x70/0x70 [ 5796.124329] ret_from_fork+0x1f/0x30 [ 5796.489915] kthread+0x1d1/0x200 [ 5796.491455] ? set_kthread_struct+0x70/0x70 [ 5796.493550] ret_from_fork+0x1f/0x30 [ 5800.930093] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3287s! [ 5800.935658] Showing busy workqueues and worker pools: [ 5800.941741] workqueue events: flags=0x0 [ 5800.947143] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5800.951494] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5800.959376] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5800.963006] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5800.967329] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5800.972874] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5800.980112] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5800.986313] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5800.991872] workqueue events_long: flags=0x0 [ 5800.995667] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5801.000312] pending: lu_objects_flush [obdclass] [ 5801.002117] workqueue events_unbound: flags=0x2 [ 5801.003367] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5801.008218] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5801.010134] workqueue events_power_efficient: flags=0x80 [ 5801.013453] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5801.016937] pending: neigh_periodic_work, do_cache_clean [ 5801.019321] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5801.024001] pending: neigh_periodic_work, check_lifetime [ 5801.027435] workqueue mm_percpu_wq: flags=0x8 [ 5801.029664] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5801.032340] pending: vmstat_update, lru_add_drain_per_cpu [ 5801.038728] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5801.048108] pending: vmstat_update [ 5801.049344] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5801.055865] pending: lru_add_drain_per_cpu BAR(48) [ 5801.060187] workqueue ipv6_addrconf: flags=0x40008 [ 5801.062452] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5801.067371] pending: addrconf_verify_work [ 5801.071852] workqueue knbd0-recv: flags=0x1a [ 5801.074854] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5801.079465] in-flight: 650:recv_work [ 5801.084203] workqueue ptlrpc_pinger: flags=0x2 [ 5801.086133] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5801.088842] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5801.091406] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3287s workers=4 idle: 36 466 29757 [ 5801.094644] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5791s workers=2 idle: 652 [ 5808.059321] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 5808.066528] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5808.087635] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5808.092070] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5808.095107] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5808.096641] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5808.103529] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5808.106453] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5808.109072] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5808.111828] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5808.114381] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5808.117217] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5808.119798] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5808.122761] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5808.124270] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5808.126755] Call Trace: [ 5808.127575] [ 5808.128217] ? show_regs.cold.9+0x22/0x2f [ 5808.129895] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5808.131341] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5808.133185] ? watchdog+0x40/0x40 [ 5808.134505] ? hrtimer_interrupt+0x128/0x2c0 [ 5808.136156] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5808.137924] ? apic_timer_interrupt+0xf/0x20 [ 5808.139517] [ 5808.140181] ? native_safe_halt+0x2e/0x50 [ 5808.141915] kvm_wait+0xcd/0xe0 [ 5808.143078] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5808.145043] do_raw_spin_lock+0x177/0x190 [ 5808.146484] _raw_spin_lock+0x21/0x30 [ 5808.148006] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5808.150402] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5808.152828] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5808.155003] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5808.157333] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5808.159540] ? woken_wake_function+0x30/0x30 [ 5808.161354] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5808.163915] kthread+0x1d1/0x200 [ 5808.165094] ? set_kthread_struct+0x70/0x70 [ 5808.166732] ret_from_fork+0x1f/0x30 [ 5816.065198] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5816.078111] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5816.117983] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5816.127228] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5816.134384] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5816.138090] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5816.156048] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5816.162578] RAX: 0000000000006ed6 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5816.169376] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5816.177346] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5816.184989] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5816.189175] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5816.204922] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5816.211299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5816.220493] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5816.226105] Call Trace: [ 5816.233904] [ 5816.234554] ? show_regs.cold.9+0x22/0x2f [ 5816.246566] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5816.248206] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5816.255196] ? watchdog+0x40/0x40 [ 5816.259116] ? hrtimer_interrupt+0x128/0x2c0 [ 5816.263545] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5816.265512] ? apic_timer_interrupt+0xf/0x20 [ 5816.271865] [ 5816.276535] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5816.281709] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5816.291296] do_raw_spin_lock+0x177/0x190 [ 5816.300845] _raw_spin_lock+0x21/0x30 [ 5816.304920] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5816.314567] ? do_raw_spin_unlock+0x75/0x190 [ 5816.320049] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5816.327290] ? do_wait_intr+0xf0/0xf0 [ 5816.333257] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5816.344241] mdc_reint+0x2d/0x110 [mdc] [ 5816.347018] mdc_unlink+0x658/0x870 [mdc] [ 5816.349280] lmv_unlink+0x34d/0xbd0 [lmv] [ 5816.356960] ll_unlink+0x24d/0x7f0 [lustre] [ 5816.368700] vfs_unlink+0x202/0x330 [ 5816.373966] do_unlinkat+0x2fa/0x460 [ 5816.380157] __x64_sys_unlink+0x37/0x50 [ 5816.388204] do_syscall_64+0xc1/0x440 [ 5816.399675] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5816.406235] RIP: 0033:0x7f83e5f5c10b [ 5816.415084] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5816.448426] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5816.458744] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5816.468702] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5816.479214] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5816.490005] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5816.498363] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5821.045056] rcu: INFO: rcu_sched self-detected stall on CPU [ 5821.048326] rcu: 0-....: (2440775 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 5821.053386] (t=3840094 jiffies g=262441 q=58207) [ 5821.054720] rcu: rcu_sched kthread starved for 3309282 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 5821.061663] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5821.066960] rcu: RCU grace-period kthread stack dump: [ 5821.069578] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5821.076808] Call Trace: [ 5821.077872] __schedule+0x351/0xcb0 [ 5821.079527] schedule+0xc0/0x180 [ 5821.081249] schedule_timeout+0xb4/0x190 [ 5821.082841] ? do_raw_spin_unlock+0x75/0x190 [ 5821.084949] ? __next_timer_interrupt+0x160/0x160 [ 5821.088962] ? prepare_to_swait_event+0xf3/0x140 [ 5821.092206] rcu_gp_kthread+0x8dd/0x15b0 [ 5821.094850] ? force_qs_rnp+0x2c0/0x2c0 [ 5821.098976] kthread+0x1d1/0x200 [ 5821.102242] ? set_kthread_struct+0x70/0x70 [ 5821.105960] ret_from_fork+0x1f/0x30 [ 5821.107552] rcu: Stack dump where RCU GP kthread last ran: [ 5821.111472] Sending NMI from CPU 0 to CPUs 2: [ 5821.114292] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [ 5821.118020] NMI backtrace for cpu 0 [ 5821.119880] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5821.124794] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5821.130673] Call Trace: [ 5821.131619] [ 5821.132874] ? dump_stack+0xbb/0x10e [ 5821.134616] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5821.137975] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5821.144489] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5821.150332] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5821.154747] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5821.157048] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5821.159863] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5821.162732] ? update_process_times+0x6d/0xc0 [ 5821.164841] ? tick_sched_handle.isra.18+0x27/0x90 [ 5821.167131] ? tick_sched_timer+0x53/0xb0 [ 5821.169616] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5821.171964] ? tick_sched_handle.isra.18+0x90/0x90 [ 5821.174298] ? hrtimer_interrupt+0x128/0x2c0 [ 5821.177305] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5821.179604] ? apic_timer_interrupt+0xf/0x20 [ 5821.182854] [ 5821.184174] ? native_safe_halt+0x2e/0x50 [ 5821.186796] ? kvm_wait+0xcd/0xe0 [ 5821.188805] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5821.191434] ? do_raw_spin_lock+0x177/0x190 [ 5821.194583] ? _raw_spin_lock+0x21/0x30 [ 5821.195952] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5821.199126] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5821.201464] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5821.203567] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5821.206469] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5821.209229] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5821.211577] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5821.214095] ? __next_timer_interrupt+0x160/0x160 [ 5821.216090] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5821.218578] ? do_wait_intr+0xf0/0xf0 [ 5821.220184] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5821.223059] ? kthread+0x1d1/0x200 [ 5821.224300] ? set_kthread_struct+0x70/0x70 [ 5821.226788] ? ret_from_fork+0x1f/0x30 [ 5831.139104] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3317s! [ 5831.143588] Showing busy workqueues and worker pools: [ 5831.147603] workqueue events: flags=0x0 [ 5831.151114] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5831.157798] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5831.171104] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5831.178862] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5831.186080] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5831.193230] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5831.203073] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5831.210873] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5831.217050] workqueue events_long: flags=0x0 [ 5831.220405] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5831.226201] pending: lu_objects_flush [obdclass] [ 5831.230734] workqueue events_unbound: flags=0x2 [ 5831.233602] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5831.237883] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5831.242062] workqueue events_power_efficient: flags=0x80 [ 5831.244995] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5831.248884] pending: neigh_periodic_work, do_cache_clean [ 5831.253063] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5831.257887] pending: neigh_periodic_work, check_lifetime [ 5831.262217] workqueue mm_percpu_wq: flags=0x8 [ 5831.265041] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5831.269690] pending: vmstat_update, lru_add_drain_per_cpu [ 5831.273207] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5831.276870] pending: vmstat_update [ 5831.279496] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5831.284567] pending: lru_add_drain_per_cpu BAR(48) [ 5831.289391] workqueue ipv6_addrconf: flags=0x40008 [ 5831.295752] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5831.304356] pending: addrconf_verify_work [ 5831.306496] workqueue knbd0-recv: flags=0x1a [ 5831.314176] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5831.319215] in-flight: 650:recv_work [ 5831.323239] workqueue ptlrpc_pinger: flags=0x2 [ 5831.328271] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5831.332259] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5831.339216] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3318s workers=4 idle: 36 466 29757 [ 5831.343915] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5821s workers=2 idle: 652 [ 5836.059067] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 5836.064094] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5836.082174] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5836.086489] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5836.090423] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5836.092073] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5836.099120] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5836.102082] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5836.104717] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5836.107778] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5836.111043] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5836.114291] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5836.117196] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5836.121088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5836.123641] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5836.126383] Call Trace: [ 5836.127318] [ 5836.128138] ? show_regs.cold.9+0x22/0x2f [ 5836.130112] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5836.131902] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5836.133707] ? watchdog+0x40/0x40 [ 5836.134990] ? hrtimer_interrupt+0x128/0x2c0 [ 5836.136964] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5836.139153] ? apic_timer_interrupt+0xf/0x20 [ 5836.140909] [ 5836.141886] ? native_safe_halt+0x2e/0x50 [ 5836.143849] kvm_wait+0xcd/0xe0 [ 5836.144952] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5836.147040] do_raw_spin_lock+0x177/0x190 [ 5836.148243] _raw_spin_lock+0x21/0x30 [ 5836.149720] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5836.152403] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5836.154487] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5836.157183] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5836.159699] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5836.162153] ? woken_wake_function+0x30/0x30 [ 5836.163987] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5836.166646] kthread+0x1d1/0x200 [ 5836.167947] ? set_kthread_struct+0x70/0x70 [ 5836.170033] ret_from_fork+0x1f/0x30 [ 5844.062038] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5844.064999] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5844.083245] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5844.086840] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5844.090089] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5844.091521] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5844.098236] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5844.101170] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5844.104017] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5844.106156] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5844.107921] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5844.109651] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5844.113882] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5844.118060] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5844.121168] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5844.124910] Call Trace: [ 5844.126005] [ 5844.127566] ? show_regs.cold.9+0x22/0x2f [ 5844.129151] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5844.137875] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5844.140166] ? watchdog+0x40/0x40 [ 5844.142663] ? hrtimer_interrupt+0x128/0x2c0 [ 5844.145091] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5844.147667] ? apic_timer_interrupt+0xf/0x20 [ 5844.149694] [ 5844.151190] ? native_safe_halt+0x2e/0x50 [ 5844.153038] kvm_wait+0xcd/0xe0 [ 5844.154431] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5844.156703] do_raw_spin_lock+0x177/0x190 [ 5844.158048] _raw_spin_lock+0x21/0x30 [ 5844.159324] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5844.162023] ? do_raw_spin_unlock+0x75/0x190 [ 5844.163686] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5844.166446] ? do_wait_intr+0xf0/0xf0 [ 5844.167716] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5844.170681] mdc_reint+0x2d/0x110 [mdc] [ 5844.172817] mdc_unlink+0x658/0x870 [mdc] [ 5844.174834] lmv_unlink+0x34d/0xbd0 [lmv] [ 5844.176558] ll_unlink+0x24d/0x7f0 [lustre] [ 5844.179182] vfs_unlink+0x202/0x330 [ 5844.180792] do_unlinkat+0x2fa/0x460 [ 5844.182476] __x64_sys_unlink+0x37/0x50 [ 5844.183755] do_syscall_64+0xc1/0x440 [ 5844.184833] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5844.187010] RIP: 0033:0x7f83e5f5c10b [ 5844.188724] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5844.198150] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5844.202115] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5844.205146] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5844.209129] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5844.212078] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5844.214943] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5848.060145] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5848.067289] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5848.093738] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5848.098216] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5848.101811] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5848.104738] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5848.106502] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5848.113779] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5848.117374] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5848.123267] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5848.129850] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5848.134665] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5848.138556] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5848.143269] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5848.148388] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5848.154950] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5848.159818] Call Trace: [ 5848.161280] [ 5848.162510] ? show_regs.cold.9+0x22/0x2f [ 5848.164890] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5848.167631] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5848.170545] ? watchdog+0x40/0x40 [ 5848.172348] ? hrtimer_interrupt+0x128/0x2c0 [ 5848.174572] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5848.177811] ? apic_timer_interrupt+0xf/0x20 [ 5848.180338] [ 5848.184083] ? native_safe_halt+0x2e/0x50 [ 5848.186377] kvm_wait+0xcd/0xe0 [ 5848.188760] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5848.191899] do_raw_spin_lock+0x177/0x190 [ 5848.195093] _raw_spin_lock+0x21/0x30 [ 5848.197634] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5848.201266] process_one_work+0x2c8/0x700 [ 5848.203440] worker_thread+0x296/0x6e0 [ 5848.205743] ? rescuer_thread+0x570/0x570 [ 5848.208299] kthread+0x1d1/0x200 [ 5848.210429] ? set_kthread_struct+0x70/0x70 [ 5848.212849] ret_from_fork+0x1f/0x30 [ 5856.056023] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 5856.059211] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5856.079958] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5856.084538] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5856.088015] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5856.090184] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5856.097374] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5856.100317] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5856.103017] RDX: 00000000000062af RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5856.105902] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5856.108727] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5856.111521] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5856.113968] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5856.116258] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5856.118792] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5856.121870] Call Trace: [ 5856.123023] [ 5856.123806] ? show_regs.cold.9+0x22/0x2f [ 5856.125517] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5856.127288] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5856.129241] ? watchdog+0x40/0x40 [ 5856.130547] ? hrtimer_interrupt+0x128/0x2c0 [ 5856.132124] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5856.134362] ? apic_timer_interrupt+0xf/0x20 [ 5856.136117] [ 5856.136957] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5856.139570] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5856.141801] do_raw_spin_lock+0x177/0x190 [ 5856.143818] _raw_spin_lock+0x21/0x30 [ 5856.145223] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5856.147621] obd_mod_free+0x46/0x140 [ptlrpc] [ 5856.149731] mdc_commit_open+0x91/0xb0 [mdc] [ 5856.152092] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5856.155187] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5856.157995] after_reply+0xe2c/0x1780 [ptlrpc] [ 5856.160295] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5856.162929] ? __next_timer_interrupt+0x160/0x160 [ 5856.164882] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5856.167472] ? do_wait_intr+0xf0/0xf0 [ 5856.168869] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5856.171477] kthread+0x1d1/0x200 [ 5856.172930] ? set_kthread_struct+0x70/0x70 [ 5856.174848] ret_from_fork+0x1f/0x30 [ 5861.858971] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3348s! [ 5861.867197] Showing busy workqueues and worker pools: [ 5861.871244] workqueue events: flags=0x0 [ 5861.873598] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5861.877975] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5861.884267] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5861.887973] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5861.892343] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5861.896486] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5861.900302] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5861.904009] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5861.908749] workqueue events_long: flags=0x0 [ 5861.909992] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5861.920501] pending: lu_objects_flush [obdclass] [ 5861.930266] workqueue events_unbound: flags=0x2 [ 5861.932937] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5861.939355] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5861.942261] workqueue events_power_efficient: flags=0x80 [ 5861.946866] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5861.949734] pending: neigh_periodic_work, do_cache_clean [ 5861.953495] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5861.957253] pending: neigh_periodic_work, check_lifetime [ 5861.959929] workqueue mm_percpu_wq: flags=0x8 [ 5861.962846] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5861.969001] pending: vmstat_update, lru_add_drain_per_cpu [ 5861.973230] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5861.975899] pending: vmstat_update [ 5861.977705] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5861.982736] pending: lru_add_drain_per_cpu BAR(48) [ 5861.986245] workqueue ipv6_addrconf: flags=0x40008 [ 5861.988962] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5861.996714] pending: addrconf_verify_work [ 5862.003215] workqueue knbd0-recv: flags=0x1a [ 5862.009893] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5862.012735] in-flight: 650:recv_work [ 5862.013938] workqueue ptlrpc_pinger: flags=0x2 [ 5862.022045] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5862.029615] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5862.031527] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3348s workers=4 idle: 36 466 29757 [ 5862.048329] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5852s workers=2 idle: 652 [ 5864.059012] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 5864.063225] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5864.111208] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5864.129013] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5864.137671] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5864.143141] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5864.151407] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5864.155792] RAX: 00000000000077a5 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5864.159573] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5864.163903] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5864.167538] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5864.171823] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5864.175398] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5864.179430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5864.182451] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5864.186709] Call Trace: [ 5864.188447] [ 5864.189561] ? show_regs.cold.9+0x22/0x2f [ 5864.191467] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5864.193695] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5864.195729] ? watchdog+0x40/0x40 [ 5864.197255] ? hrtimer_interrupt+0x128/0x2c0 [ 5864.199876] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5864.202100] ? apic_timer_interrupt+0xf/0x20 [ 5864.209855] [ 5864.210791] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5864.217644] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5864.222440] do_raw_spin_lock+0x177/0x190 [ 5864.225696] _raw_spin_lock+0x21/0x30 [ 5864.228951] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5864.236128] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5864.243116] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5864.250583] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5864.257883] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5864.262570] ? woken_wake_function+0x30/0x30 [ 5864.264695] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5864.270098] kthread+0x1d1/0x200 [ 5864.271590] ? set_kthread_struct+0x70/0x70 [ 5864.277732] ret_from_fork+0x1f/0x30 [ 5872.062312] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5872.066800] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5872.092973] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5872.099390] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5872.104356] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5872.107089] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5872.118119] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5872.121533] RAX: 0000000000000005 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5872.125177] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5872.128124] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5872.131106] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5872.134352] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5872.136759] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5872.139970] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5872.142707] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5872.146295] Call Trace: [ 5872.147313] [ 5872.148136] ? show_regs.cold.9+0x22/0x2f [ 5872.150043] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5872.152081] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5872.154011] ? watchdog+0x40/0x40 [ 5872.155478] ? hrtimer_interrupt+0x128/0x2c0 [ 5872.157314] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5872.160771] ? apic_timer_interrupt+0xf/0x20 [ 5872.164157] [ 5872.165815] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5872.167642] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5872.168842] do_raw_spin_lock+0x177/0x190 [ 5872.172122] _raw_spin_lock+0x21/0x30 [ 5872.174657] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5872.177370] ? do_raw_spin_unlock+0x75/0x190 [ 5872.179585] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5872.183226] ? do_wait_intr+0xf0/0xf0 [ 5872.186375] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5872.189739] mdc_reint+0x2d/0x110 [mdc] [ 5872.191765] mdc_unlink+0x658/0x870 [mdc] [ 5872.193727] lmv_unlink+0x34d/0xbd0 [lmv] [ 5872.195708] ll_unlink+0x24d/0x7f0 [lustre] [ 5872.198128] vfs_unlink+0x202/0x330 [ 5872.199641] do_unlinkat+0x2fa/0x460 [ 5872.200875] __x64_sys_unlink+0x37/0x50 [ 5872.201937] do_syscall_64+0xc1/0x440 [ 5872.203323] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5872.204600] RIP: 0033:0x7f83e5f5c10b [ 5872.205554] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5872.213711] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5872.218624] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5872.222622] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5872.227109] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5872.231519] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5872.237046] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5876.060014] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5876.064565] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5876.086302] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5876.092016] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5876.095930] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5876.099382] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5876.101819] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5876.114557] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5876.121472] RAX: 0000000000002972 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5876.125123] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5876.129720] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5876.138304] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5876.142465] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5876.145746] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5876.152608] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5876.155720] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5876.161382] Call Trace: [ 5876.162312] [ 5876.163026] ? show_regs.cold.9+0x22/0x2f [ 5876.164596] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5876.166494] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5876.168223] ? watchdog+0x40/0x40 [ 5876.169389] ? hrtimer_interrupt+0x128/0x2c0 [ 5876.170814] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5876.172886] ? apic_timer_interrupt+0xf/0x20 [ 5876.175016] [ 5876.176277] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5876.179017] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5876.181799] do_raw_spin_lock+0x177/0x190 [ 5876.183560] _raw_spin_lock+0x21/0x30 [ 5876.185367] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5876.187591] process_one_work+0x2c8/0x700 [ 5876.189006] worker_thread+0x296/0x6e0 [ 5876.190397] ? rescuer_thread+0x570/0x570 [ 5876.192908] kthread+0x1d1/0x200 [ 5876.194071] ? set_kthread_struct+0x70/0x70 [ 5876.195579] ret_from_fork+0x1f/0x30 [ 5888.056009] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 5888.061272] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5888.088760] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5888.102937] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5888.109904] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5888.112450] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5888.129608] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5888.137264] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5888.149735] RDX: 0000000000000662 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5888.159723] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5888.166493] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5888.174450] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5888.180349] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5888.185291] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5888.188507] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5888.194170] Call Trace: [ 5888.195376] [ 5888.196601] ? show_regs.cold.9+0x22/0x2f [ 5888.203723] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5888.206211] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5888.212002] ? watchdog+0x40/0x40 [ 5888.214402] ? hrtimer_interrupt+0x128/0x2c0 [ 5888.220468] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5888.226056] ? apic_timer_interrupt+0xf/0x20 [ 5888.231846] [ 5888.236075] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5888.240405] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5888.247095] do_raw_spin_lock+0x177/0x190 [ 5888.248398] _raw_spin_lock+0x21/0x30 [ 5888.260522] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5888.266777] obd_mod_free+0x46/0x140 [ptlrpc] [ 5888.272132] mdc_commit_open+0x91/0xb0 [mdc] [ 5888.274414] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5888.281018] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5888.286564] after_reply+0xe2c/0x1780 [ptlrpc] [ 5888.294492] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5888.297182] ? __next_timer_interrupt+0x160/0x160 [ 5888.299485] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5888.306379] ? do_wait_intr+0xf0/0xf0 [ 5888.309110] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5888.315149] kthread+0x1d1/0x200 [ 5888.317286] ? set_kthread_struct+0x70/0x70 [ 5888.321251] ret_from_fork+0x1f/0x30 [ 5892.059026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 5892.061943] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5892.109413] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5892.125627] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5892.138961] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5892.143900] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5892.159199] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5892.163473] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5892.170231] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5892.176594] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5892.184465] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5892.191846] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5892.199898] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5892.207364] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5892.215939] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5892.227619] Call Trace: [ 5892.231372] [ 5892.232501] ? show_regs.cold.9+0x22/0x2f [ 5892.236543] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5892.242159] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5892.247112] ? watchdog+0x40/0x40 [ 5892.251184] ? hrtimer_interrupt+0x128/0x2c0 [ 5892.252686] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5892.258779] ? apic_timer_interrupt+0xf/0x20 [ 5892.260213] [ 5892.260943] ? native_safe_halt+0x2e/0x50 [ 5892.267957] kvm_wait+0xcd/0xe0 [ 5892.272413] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5892.275434] do_raw_spin_lock+0x177/0x190 [ 5892.278751] _raw_spin_lock+0x21/0x30 [ 5892.283534] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5892.286754] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5892.292311] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5892.296280] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5892.298767] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5892.303128] ? woken_wake_function+0x30/0x30 [ 5892.305297] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5892.314248] kthread+0x1d1/0x200 [ 5892.315475] ? set_kthread_struct+0x70/0x70 [ 5892.321132] ret_from_fork+0x1f/0x30 [ 5892.322561] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3379s! [ 5892.325901] Showing busy workqueues and worker pools: [ 5892.334913] workqueue events: flags=0x0 [ 5892.338460] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5892.341584] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5892.352277] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5892.355957] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5892.360211] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5892.366221] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5892.371429] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5892.384154] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5892.397371] workqueue events_long: flags=0x0 [ 5892.401668] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5892.407297] pending: lu_objects_flush [obdclass] [ 5892.411567] workqueue events_unbound: flags=0x2 [ 5892.416368] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5892.419887] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5892.422698] workqueue events_power_efficient: flags=0x80 [ 5892.425359] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5892.428278] pending: neigh_periodic_work, do_cache_clean [ 5892.430581] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5892.433604] pending: neigh_periodic_work, check_lifetime [ 5892.436357] workqueue mm_percpu_wq: flags=0x8 [ 5892.438453] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5892.441497] pending: vmstat_update, lru_add_drain_per_cpu [ 5892.443980] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5892.446447] pending: vmstat_update [ 5892.447943] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5892.450513] pending: lru_add_drain_per_cpu BAR(48) [ 5892.452625] workqueue ipv6_addrconf: flags=0x40008 [ 5892.455517] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5892.465411] pending: addrconf_verify_work [ 5892.468692] workqueue knbd0-recv: flags=0x1a [ 5892.472416] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5892.474778] in-flight: 650:recv_work [ 5892.479769] workqueue ptlrpc_pinger: flags=0x2 [ 5892.483514] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5892.489971] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5892.498118] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3379s workers=4 idle: 36 466 29757 [ 5892.509197] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5882s workers=2 idle: 652 [ 5900.062024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5900.067814] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5900.090316] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5900.096397] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5900.107232] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5900.110346] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5900.122242] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5900.127225] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5900.132180] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5900.138363] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5900.142010] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5900.146575] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5900.152085] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5900.162373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5900.168475] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5900.180104] Call Trace: [ 5900.185364] [ 5900.186466] ? show_regs.cold.9+0x22/0x2f [ 5900.190066] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5900.194205] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5900.198968] ? watchdog+0x40/0x40 [ 5900.201346] ? hrtimer_interrupt+0x128/0x2c0 [ 5900.203775] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5900.210453] ? apic_timer_interrupt+0xf/0x20 [ 5900.216315] [ 5900.219252] ? native_safe_halt+0x2e/0x50 [ 5900.223631] kvm_wait+0xcd/0xe0 [ 5900.226853] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5900.231398] do_raw_spin_lock+0x177/0x190 [ 5900.233196] _raw_spin_lock+0x21/0x30 [ 5900.235290] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5900.238440] ? do_raw_spin_unlock+0x75/0x190 [ 5900.241082] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5900.244053] ? do_wait_intr+0xf0/0xf0 [ 5900.246263] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5900.250276] mdc_reint+0x2d/0x110 [mdc] [ 5900.252782] mdc_unlink+0x658/0x870 [mdc] [ 5900.254885] lmv_unlink+0x34d/0xbd0 [lmv] [ 5900.257511] ll_unlink+0x24d/0x7f0 [lustre] [ 5900.259980] vfs_unlink+0x202/0x330 [ 5900.261832] do_unlinkat+0x2fa/0x460 [ 5900.263618] __x64_sys_unlink+0x37/0x50 [ 5900.266358] do_syscall_64+0xc1/0x440 [ 5900.268222] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5900.270501] RIP: 0033:0x7f83e5f5c10b [ 5900.273333] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5900.283225] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5900.286628] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5900.290687] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5900.294397] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5900.298288] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5900.302257] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5904.060042] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5904.065124] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5904.091352] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5904.095625] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5904.098656] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5904.101826] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5904.103845] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5904.110608] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5904.113826] RAX: 0000000000007a97 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5904.116319] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5904.118850] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5904.121215] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5904.123998] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5904.126488] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5904.130077] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5904.132147] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5904.134640] Call Trace: [ 5904.135633] [ 5904.137024] ? show_regs.cold.9+0x22/0x2f [ 5904.138322] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5904.140150] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5904.141871] ? watchdog+0x40/0x40 [ 5904.143491] ? hrtimer_interrupt+0x128/0x2c0 [ 5904.145143] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5904.147216] ? apic_timer_interrupt+0xf/0x20 [ 5904.148832] [ 5904.149555] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5904.151336] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5904.153614] do_raw_spin_lock+0x177/0x190 [ 5904.156102] _raw_spin_lock+0x21/0x30 [ 5904.157559] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5904.160143] process_one_work+0x2c8/0x700 [ 5904.161495] worker_thread+0x296/0x6e0 [ 5904.162621] ? rescuer_thread+0x570/0x570 [ 5904.164226] kthread+0x1d1/0x200 [ 5904.165234] ? set_kthread_struct+0x70/0x70 [ 5904.167005] ret_from_fork+0x1f/0x30 [ 5920.059011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 5920.059200] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 5920.066985] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5920.067063] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5920.067180] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5920.067183] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5920.067197] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5920.067201] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5920.067206] RAX: 0000000000002bb7 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5920.067208] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5920.067210] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5920.067212] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5920.067215] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5920.067218] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5920.067221] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5920.067223] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5920.067231] Call Trace: [ 5920.067236] [ 5920.067241] ? show_regs.cold.9+0x22/0x2f [ 5920.067249] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5920.067256] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5920.067260] ? watchdog+0x40/0x40 [ 5920.067265] ? hrtimer_interrupt+0x128/0x2c0 [ 5920.067268] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5920.067277] ? apic_timer_interrupt+0xf/0x20 [ 5920.067282] [ 5920.067285] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5920.067290] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5920.067296] do_raw_spin_lock+0x177/0x190 [ 5920.067300] _raw_spin_lock+0x21/0x30 [ 5920.067305] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5920.067880] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5920.068220] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5920.068700] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5920.069028] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5920.069406] ? woken_wake_function+0x30/0x30 [ 5920.069479] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5920.069712] kthread+0x1d1/0x200 [ 5920.069717] ? set_kthread_struct+0x70/0x70 [ 5920.069779] ret_from_fork+0x1f/0x30 [ 5920.230952] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5920.255270] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5920.261029] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5920.265285] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5920.268635] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5920.279309] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5920.283367] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5920.287860] RDX: 0000000000007bd8 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5920.291702] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5920.295663] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5920.299577] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5920.303040] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5920.307266] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5920.310530] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5920.314530] Call Trace: [ 5920.315916] [ 5920.317310] ? show_regs.cold.9+0x22/0x2f [ 5920.319199] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5920.322112] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5920.324372] ? watchdog+0x40/0x40 [ 5920.326180] ? hrtimer_interrupt+0x128/0x2c0 [ 5920.328707] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5920.331190] ? apic_timer_interrupt+0xf/0x20 [ 5920.333594] [ 5920.334691] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5920.337602] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5920.339359] do_raw_spin_lock+0x177/0x190 [ 5920.340943] _raw_spin_lock+0x21/0x30 [ 5920.342778] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5920.345938] obd_mod_free+0x46/0x140 [ptlrpc] [ 5920.348499] mdc_commit_open+0x91/0xb0 [mdc] [ 5920.350291] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5920.353133] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5920.356031] after_reply+0xe2c/0x1780 [ptlrpc] [ 5920.358735] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5920.361779] ? __next_timer_interrupt+0x160/0x160 [ 5920.364288] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5920.367368] ? do_wait_intr+0xf0/0xf0 [ 5920.369634] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5920.372693] kthread+0x1d1/0x200 [ 5920.374499] ? set_kthread_struct+0x70/0x70 [ 5920.376880] ret_from_fork+0x1f/0x30 [ 5922.784532] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3409s! [ 5922.790840] Showing busy workqueues and worker pools: [ 5922.792758] workqueue events: flags=0x0 [ 5922.797757] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5922.806928] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5922.816622] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5922.824813] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5922.833120] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5922.839335] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5922.842879] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5922.845809] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5922.850754] workqueue events_long: flags=0x0 [ 5922.852620] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5922.855196] pending: lu_objects_flush [obdclass] [ 5922.858034] workqueue events_unbound: flags=0x2 [ 5922.859627] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5922.862584] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5922.865212] workqueue events_power_efficient: flags=0x80 [ 5922.867300] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5922.869945] pending: neigh_periodic_work, do_cache_clean [ 5922.871922] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5922.874906] pending: neigh_periodic_work, check_lifetime [ 5922.877641] workqueue mm_percpu_wq: flags=0x8 [ 5922.880522] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5922.888173] pending: vmstat_update, lru_add_drain_per_cpu [ 5922.892661] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5922.900709] pending: vmstat_update [ 5922.908690] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5922.916435] pending: lru_add_drain_per_cpu BAR(48) [ 5922.923886] workqueue ipv6_addrconf: flags=0x40008 [ 5922.931862] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5922.936112] pending: addrconf_verify_work [ 5922.943292] workqueue knbd0-recv: flags=0x1a [ 5922.945129] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5922.949291] in-flight: 650:recv_work [ 5922.957114] workqueue ptlrpc_pinger: flags=0x2 [ 5922.959210] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5922.965719] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5922.971711] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3409s workers=4 idle: 36 466 29757 [ 5922.981086] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5913s workers=2 idle: 652 [ 5928.062010] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 5928.066030] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5928.091392] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5928.095886] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5928.099390] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5928.102153] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5928.110102] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5928.113341] RAX: 0000000000002191 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5928.116383] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5928.119347] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5928.122329] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5928.125618] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5928.129368] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5928.132875] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5928.135465] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5928.138793] Call Trace: [ 5928.139695] [ 5928.140479] ? show_regs.cold.9+0x22/0x2f [ 5928.142559] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5928.144664] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5928.146527] ? watchdog+0x40/0x40 [ 5928.148012] ? hrtimer_interrupt+0x128/0x2c0 [ 5928.149929] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5928.151949] ? apic_timer_interrupt+0xf/0x20 [ 5928.153878] [ 5928.155273] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5928.158212] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5928.160264] do_raw_spin_lock+0x177/0x190 [ 5928.162159] _raw_spin_lock+0x21/0x30 [ 5928.163628] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5928.166226] ? do_raw_spin_unlock+0x75/0x190 [ 5928.168279] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5928.170995] ? do_wait_intr+0xf0/0xf0 [ 5928.172599] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5928.175808] mdc_reint+0x2d/0x110 [mdc] [ 5928.177284] mdc_unlink+0x658/0x870 [mdc] [ 5928.179571] lmv_unlink+0x34d/0xbd0 [lmv] [ 5928.181633] ll_unlink+0x24d/0x7f0 [lustre] [ 5928.184275] vfs_unlink+0x202/0x330 [ 5928.186242] do_unlinkat+0x2fa/0x460 [ 5928.187770] __x64_sys_unlink+0x37/0x50 [ 5928.189192] do_syscall_64+0xc1/0x440 [ 5928.190893] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5928.192962] RIP: 0033:0x7f83e5f5c10b [ 5928.194806] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5928.206163] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5928.209829] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5928.214188] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5928.217570] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5928.222179] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5928.234807] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5932.071086] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5932.079522] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5932.105908] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5932.115592] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5932.120718] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5932.124117] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5932.127089] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5932.138180] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5932.142340] RAX: 0000000000006707 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5932.148029] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5932.151257] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5932.154169] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5932.157204] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5932.160182] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5932.165482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5932.168015] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5932.171682] Call Trace: [ 5932.172565] [ 5932.173512] ? show_regs.cold.9+0x22/0x2f [ 5932.175656] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5932.178026] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5932.181158] ? watchdog+0x40/0x40 [ 5932.183100] ? hrtimer_interrupt+0x128/0x2c0 [ 5932.185502] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5932.187880] ? apic_timer_interrupt+0xf/0x20 [ 5932.190794] [ 5932.191614] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5932.195107] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5932.198214] do_raw_spin_lock+0x177/0x190 [ 5932.201267] _raw_spin_lock+0x21/0x30 [ 5932.203599] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5932.206810] process_one_work+0x2c8/0x700 [ 5932.208697] worker_thread+0x296/0x6e0 [ 5932.211787] ? rescuer_thread+0x570/0x570 [ 5932.214938] kthread+0x1d1/0x200 [ 5932.217422] ? set_kthread_struct+0x70/0x70 [ 5932.220463] ret_from_fork+0x1f/0x30 [ 5948.056016] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 5948.059163] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 5948.064400] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common [ 5948.067966] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5948.068039] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5948.068045] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5948.068048] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5948.068080] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5948.068084] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5948.068089] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5948.068092] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5948.068094] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5948.068096] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5948.068098] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5948.068101] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5948.068104] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5948.068106] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5948.068114] Call Trace: [ 5948.068129] [ 5948.068134] ? show_regs.cold.9+0x22/0x2f [ 5948.068143] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5948.068227] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5948.068231] ? watchdog+0x40/0x40 [ 5948.068235] ? hrtimer_interrupt+0x128/0x2c0 [ 5948.068239] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5948.068244] ? apic_timer_interrupt+0xf/0x20 [ 5948.068262] [ 5948.068265] ? native_safe_halt+0x2e/0x50 [ 5948.068269] kvm_wait+0xcd/0xe0 [ 5948.068276] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5948.068284] do_raw_spin_lock+0x177/0x190 [ 5948.068288] _raw_spin_lock+0x21/0x30 [ 5948.068292] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5948.069882] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5948.070708] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5948.070932] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5948.071191] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5948.091888] sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5948.091926] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5948.091932] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5948.091935] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5948.092158] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5948.092257] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5948.092262] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5948.092265] RDX: 0000000000005053 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5948.092267] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5948.092269] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5948.092272] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5948.092275] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5948.092278] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5948.092280] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5948.092288] Call Trace: [ 5948.092292] [ 5948.092297] ? show_regs.cold.9+0x22/0x2f [ 5948.092305] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5948.092311] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5948.092315] ? watchdog+0x40/0x40 [ 5948.092319] ? hrtimer_interrupt+0x128/0x2c0 [ 5948.092323] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5948.092330] ? apic_timer_interrupt+0xf/0x20 [ 5948.092334] [ 5948.092337] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5948.092342] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5948.092345] do_raw_spin_lock+0x177/0x190 [ 5948.092349] _raw_spin_lock+0x21/0x30 [ 5948.092356] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5948.092932] obd_mod_free+0x46/0x140 [ptlrpc] [ 5948.093286] mdc_commit_open+0x91/0xb0 [mdc] [ 5948.093330] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5948.093820] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5948.094033] after_reply+0xe2c/0x1780 [ptlrpc] [ 5948.094179] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5948.094884] ? __next_timer_interrupt+0x160/0x160 [ 5948.094893] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5948.095203] ? do_wait_intr+0xf0/0xf0 [ 5948.095211] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5948.095430] kthread+0x1d1/0x200 [ 5948.095442] ? set_kthread_struct+0x70/0x70 [ 5948.095445] ret_from_fork+0x1f/0x30 [ 5948.362555] ? woken_wake_function+0x30/0x30 [ 5948.365071] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5948.369678] kthread+0x1d1/0x200 [ 5948.374851] ? set_kthread_struct+0x70/0x70 [ 5948.376866] ret_from_fork+0x1f/0x30 [ 5952.992182] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3439s! [ 5952.996408] Showing busy workqueues and worker pools: [ 5952.998508] workqueue events: flags=0x0 [ 5953.000449] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5953.002665] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5953.007015] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5953.010464] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5953.013382] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5953.016434] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5953.020151] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5953.022896] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5953.026642] workqueue events_long: flags=0x0 [ 5953.028790] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5953.032226] pending: lu_objects_flush [obdclass] [ 5953.034275] workqueue events_unbound: flags=0x2 [ 5953.036168] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5953.038975] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5953.041355] workqueue events_power_efficient: flags=0x80 [ 5953.043860] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5953.046845] pending: neigh_periodic_work, do_cache_clean [ 5953.049384] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5953.052287] pending: neigh_periodic_work, check_lifetime [ 5953.054682] workqueue mm_percpu_wq: flags=0x8 [ 5953.056645] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5953.059393] pending: vmstat_update, lru_add_drain_per_cpu [ 5953.062068] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5953.064869] pending: vmstat_update [ 5953.066667] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5953.069150] pending: lru_add_drain_per_cpu BAR(48) [ 5953.071316] workqueue ipv6_addrconf: flags=0x40008 [ 5953.073290] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5953.076133] pending: addrconf_verify_work [ 5953.077736] workqueue knbd0-recv: flags=0x1a [ 5953.079690] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5953.082049] in-flight: 650:recv_work [ 5953.083759] workqueue ptlrpc_pinger: flags=0x2 [ 5953.085639] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5953.088119] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5953.090884] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3439s workers=4 idle: 36 466 29757 [ 5953.094585] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5943s workers=2 idle: 652 [ 5956.062200] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 5956.068712] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5956.108886] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5956.115912] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5956.121782] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5956.123189] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5956.129137] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5956.133951] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5956.136725] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5956.139987] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5956.144139] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5956.147183] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5956.150613] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5956.155050] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5956.158885] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5956.162862] Call Trace: [ 5956.163926] [ 5956.165347] ? show_regs.cold.9+0x22/0x2f [ 5956.167881] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5956.169914] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5956.172433] ? watchdog+0x40/0x40 [ 5956.173640] ? hrtimer_interrupt+0x128/0x2c0 [ 5956.175541] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5956.177862] ? apic_timer_interrupt+0xf/0x20 [ 5956.180026] [ 5956.181000] ? native_safe_halt+0x2e/0x50 [ 5956.183382] kvm_wait+0xcd/0xe0 [ 5956.184593] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5956.186059] do_raw_spin_lock+0x177/0x190 [ 5956.187046] _raw_spin_lock+0x21/0x30 [ 5956.188006] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5956.189709] ? do_raw_spin_unlock+0x75/0x190 [ 5956.190913] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5956.192670] ? do_wait_intr+0xf0/0xf0 [ 5956.193615] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5956.194905] mdc_reint+0x2d/0x110 [mdc] [ 5956.195773] mdc_unlink+0x658/0x870 [mdc] [ 5956.196708] lmv_unlink+0x34d/0xbd0 [lmv] [ 5956.197680] ll_unlink+0x24d/0x7f0 [lustre] [ 5956.198907] vfs_unlink+0x202/0x330 [ 5956.199792] do_unlinkat+0x2fa/0x460 [ 5956.200665] __x64_sys_unlink+0x37/0x50 [ 5956.201624] do_syscall_64+0xc1/0x440 [ 5956.202639] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5956.204063] RIP: 0033:0x7f83e5f5c10b [ 5956.205364] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5956.213156] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5956.216098] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5956.218879] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5956.221662] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5956.224443] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5956.227542] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5960.066970] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5960.076862] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5960.153779] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5960.165461] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5960.175238] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5960.185660] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5960.195995] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5960.216917] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5960.228968] RAX: 0000000000001189 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5960.236118] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5960.247416] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5960.259162] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5960.265637] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5960.273449] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5960.290790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5960.296744] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5960.304554] Call Trace: [ 5960.306666] [ 5960.310820] ? show_regs.cold.9+0x22/0x2f [ 5960.321108] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5960.332127] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5960.335211] ? watchdog+0x40/0x40 [ 5960.340168] ? hrtimer_interrupt+0x128/0x2c0 [ 5960.341777] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5960.351636] ? apic_timer_interrupt+0xf/0x20 [ 5960.355708] [ 5960.356651] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5960.367743] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5960.375486] do_raw_spin_lock+0x177/0x190 [ 5960.387599] _raw_spin_lock+0x21/0x30 [ 5960.389695] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5960.398345] process_one_work+0x2c8/0x700 [ 5960.401411] worker_thread+0x296/0x6e0 [ 5960.407996] ? rescuer_thread+0x570/0x570 [ 5960.411677] kthread+0x1d1/0x200 [ 5960.420563] ? set_kthread_struct+0x70/0x70 [ 5960.427718] ret_from_fork+0x1f/0x30 [ 5976.059020] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 5976.064628] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5976.102662] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5976.110091] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5976.118539] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5976.126023] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5976.138130] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5976.142254] RAX: 00000000000054a3 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5976.145986] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 5976.153109] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5976.156618] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 5976.162585] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 5976.167103] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 5976.170996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5976.174057] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 5976.180525] Call Trace: [ 5976.186297] [ 5976.187767] ? show_regs.cold.9+0x22/0x2f [ 5976.193053] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5976.196519] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5976.198850] ? watchdog+0x40/0x40 [ 5976.204616] ? hrtimer_interrupt+0x128/0x2c0 [ 5976.207772] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5976.211383] ? apic_timer_interrupt+0xf/0x20 [ 5976.213724] [ 5976.214838] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5976.217974] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5976.221246] do_raw_spin_lock+0x177/0x190 [ 5976.226661] _raw_spin_lock+0x21/0x30 [ 5976.228950] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5976.235824] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5976.238443] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5976.242523] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5976.246827] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5976.250776] ? woken_wake_function+0x30/0x30 [ 5976.252733] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5976.260691] kthread+0x1d1/0x200 [ 5976.261829] ? set_kthread_struct+0x70/0x70 [ 5976.264984] ret_from_fork+0x1f/0x30 [ 5980.061139] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 5980.071925] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5980.111686] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5980.124632] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5980.130064] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5980.138136] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5980.158241] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5980.161611] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 5980.165070] RDX: 0000000000006619 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 5980.168119] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 5980.171128] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 5980.174298] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5980.177378] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 5980.180812] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5980.183359] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 5980.186680] Call Trace: [ 5980.189257] [ 5980.189980] ? show_regs.cold.9+0x22/0x2f [ 5980.193057] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5980.195230] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5980.200482] ? watchdog+0x40/0x40 [ 5980.201580] ? hrtimer_interrupt+0x128/0x2c0 [ 5980.203750] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5980.209388] ? apic_timer_interrupt+0xf/0x20 [ 5980.211290] [ 5980.214947] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5980.218362] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5980.226316] do_raw_spin_lock+0x177/0x190 [ 5980.233928] _raw_spin_lock+0x21/0x30 [ 5980.238671] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5980.245669] obd_mod_free+0x46/0x140 [ptlrpc] [ 5980.247947] mdc_commit_open+0x91/0xb0 [mdc] [ 5980.251697] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5980.254726] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5980.260270] after_reply+0xe2c/0x1780 [ptlrpc] [ 5980.266559] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5980.274099] ? __next_timer_interrupt+0x160/0x160 [ 5980.276131] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5980.290874] ? do_wait_intr+0xf0/0xf0 [ 5980.292356] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5980.310632] kthread+0x1d1/0x200 [ 5980.311682] ? set_kthread_struct+0x70/0x70 [ 5980.319848] ret_from_fork+0x1f/0x30 [ 5983.200209] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3470s! [ 5983.208094] Showing busy workqueues and worker pools: [ 5983.224424] workqueue events: flags=0x0 [ 5983.232624] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5983.243791] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 5983.251608] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5983.269944] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 5983.283891] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5983.295408] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 5983.306410] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5983.311089] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 5983.320242] workqueue events_long: flags=0x0 [ 5983.324642] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5983.334457] pending: lu_objects_flush [obdclass] [ 5983.341416] workqueue events_unbound: flags=0x2 [ 5983.344198] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5983.352298] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5983.360564] workqueue events_power_efficient: flags=0x80 [ 5983.368216] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5983.377891] pending: neigh_periodic_work, do_cache_clean [ 5983.384117] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5983.389046] pending: neigh_periodic_work, check_lifetime [ 5983.392919] workqueue mm_percpu_wq: flags=0x8 [ 5983.395423] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5983.398495] pending: vmstat_update, lru_add_drain_per_cpu [ 5983.413394] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5983.421930] pending: vmstat_update [ 5983.426501] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 5983.432026] pending: lru_add_drain_per_cpu BAR(48) [ 5983.437421] workqueue ipv6_addrconf: flags=0x40008 [ 5983.441803] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5983.447898] pending: addrconf_verify_work [ 5983.457015] workqueue knbd0-recv: flags=0x1a [ 5983.462799] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5983.474122] in-flight: 650:recv_work [ 5983.485564] workqueue ptlrpc_pinger: flags=0x2 [ 5983.492706] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5983.498620] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 5983.504309] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3470s workers=4 idle: 36 466 29757 [ 5983.521761] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5973s workers=2 idle: 652 [ 5984.063563] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 5984.066959] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5984.117594] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5984.127880] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5984.137833] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5984.141666] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5984.159606] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5984.168629] RAX: 0000000000007f96 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5984.175365] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 5984.193742] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5984.201221] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 5984.226213] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 5984.239167] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 5984.250608] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5984.261086] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 5984.278139] Call Trace: [ 5984.279043] [ 5984.280312] ? show_regs.cold.9+0x22/0x2f [ 5984.285348] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5984.289699] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5984.295855] ? watchdog+0x40/0x40 [ 5984.297708] ? hrtimer_interrupt+0x128/0x2c0 [ 5984.301419] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5984.309544] ? apic_timer_interrupt+0xf/0x20 [ 5984.313479] [ 5984.315213] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5984.318317] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5984.321698] do_raw_spin_lock+0x177/0x190 [ 5984.323061] _raw_spin_lock+0x21/0x30 [ 5984.325268] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5984.327439] ? do_raw_spin_unlock+0x75/0x190 [ 5984.335701] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5984.344320] ? do_wait_intr+0xf0/0xf0 [ 5984.347448] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5984.356563] mdc_reint+0x2d/0x110 [mdc] [ 5984.362044] mdc_unlink+0x658/0x870 [mdc] [ 5984.369608] lmv_unlink+0x34d/0xbd0 [lmv] [ 5984.373498] ll_unlink+0x24d/0x7f0 [lustre] [ 5984.379787] vfs_unlink+0x202/0x330 [ 5984.384613] do_unlinkat+0x2fa/0x460 [ 5984.385984] __x64_sys_unlink+0x37/0x50 [ 5984.395962] do_syscall_64+0xc1/0x440 [ 5984.397370] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5984.406363] RIP: 0033:0x7f83e5f5c10b [ 5984.408302] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 5984.436345] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 5984.443979] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 5984.448961] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 5984.454621] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 5984.458949] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 5984.463286] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 5988.065136] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 5988.076167] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5988.136436] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5988.146237] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5988.156626] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5988.165419] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5988.176497] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5988.211957] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5988.224678] RAX: 0000000000007096 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 5988.232470] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 5988.246077] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 5988.254420] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 5988.263620] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 5988.272054] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 5988.289186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5988.296340] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 5988.309021] Call Trace: [ 5988.309939] [ 5988.311479] ? show_regs.cold.9+0x22/0x2f [ 5988.320525] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5988.323130] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5988.325401] ? watchdog+0x40/0x40 [ 5988.327073] ? hrtimer_interrupt+0x128/0x2c0 [ 5988.328817] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5988.335346] ? apic_timer_interrupt+0xf/0x20 [ 5988.339490] [ 5988.341954] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5988.346585] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5988.348708] do_raw_spin_lock+0x177/0x190 [ 5988.351547] _raw_spin_lock+0x21/0x30 [ 5988.356300] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5988.361547] process_one_work+0x2c8/0x700 [ 5988.367456] worker_thread+0x296/0x6e0 [ 5988.368894] ? rescuer_thread+0x570/0x570 [ 5988.375672] kthread+0x1d1/0x200 [ 5988.378541] ? set_kthread_struct+0x70/0x70 [ 5988.384632] ret_from_fork+0x1f/0x30 [ 6001.051115] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 6001.055216] rcu: 0-....: (2556478 ticks this GP) idle=30a/1/0x4000000000000000 softirq=128519/128519 fqs=126395 [ 6001.060076] (detected by 2, t=4020100 jiffies, g=262441, q=58300) [ 6001.063643] Sending NMI from CPU 2 to CPUs 0: [ 6001.065949] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 6001.066930] rcu: rcu_sched kthread starved for 3489294 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 6001.073810] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6001.077997] rcu: RCU grace-period kthread stack dump: [ 6001.080473] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6001.088071] Call Trace: [ 6001.089546] __schedule+0x351/0xcb0 [ 6001.091564] schedule+0xc0/0x180 [ 6001.093523] schedule_timeout+0xb4/0x190 [ 6001.095491] ? do_raw_spin_unlock+0x75/0x190 [ 6001.099421] ? __next_timer_interrupt+0x160/0x160 [ 6001.102248] ? prepare_to_swait_event+0xf3/0x140 [ 6001.104909] rcu_gp_kthread+0x8dd/0x15b0 [ 6001.107189] ? force_qs_rnp+0x2c0/0x2c0 [ 6001.114836] kthread+0x1d1/0x200 [ 6001.124498] ? set_kthread_struct+0x70/0x70 [ 6001.137365] ret_from_fork+0x1f/0x30 [ 6001.145579] rcu: Stack dump where RCU GP kthread last ran: [ 6001.164936] NMI backtrace for cpu 2 [ 6001.170498] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6001.178620] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6001.200949] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6001.204855] Call Trace: [ 6001.205787] [ 6001.206563] ? dump_stack+0xbb/0x10e [ 6001.208044] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6001.210168] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6001.212342] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6001.214644] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6001.216861] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6001.219115] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 6001.221328] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [ 6001.223319] ? update_process_times+0x6d/0xc0 [ 6001.225205] ? tick_sched_handle.isra.18+0x27/0x90 [ 6001.227316] ? tick_sched_timer+0x53/0xb0 [ 6001.229003] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6001.230919] ? tick_sched_handle.isra.18+0x90/0x90 [ 6001.233150] ? hrtimer_interrupt+0x128/0x2c0 [ 6001.234963] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6001.236796] ? apic_timer_interrupt+0xf/0x20 [ 6001.238528] [ 6001.239377] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6001.241652] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6001.243777] ? do_raw_spin_lock+0x177/0x190 [ 6001.245544] ? _raw_spin_lock+0x21/0x30 [ 6001.247144] ? ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6001.249621] ? process_one_work+0x2c8/0x700 [ 6001.251333] ? worker_thread+0x296/0x6e0 [ 6001.252866] ? rescuer_thread+0x570/0x570 [ 6001.254567] ? kthread+0x1d1/0x200 [ 6001.255585] ? set_kthread_struct+0x70/0x70 [ 6001.257153] ? ret_from_fork+0x1f/0x30 [ 6004.060988] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 6004.071866] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6004.139594] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6004.166687] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6004.179817] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6004.192320] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6004.214327] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6004.225077] RAX: 0000000000007412 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6004.232940] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6004.241168] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6004.250142] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6004.256058] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6004.263563] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6004.275700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6004.281999] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6004.294176] Call Trace: [ 6004.296257] [ 6004.298363] ? show_regs.cold.9+0x22/0x2f [ 6004.308110] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6004.310469] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6004.312633] ? watchdog+0x40/0x40 [ 6004.318009] ? hrtimer_interrupt+0x128/0x2c0 [ 6004.320646] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6004.327248] ? apic_timer_interrupt+0xf/0x20 [ 6004.329209] [ 6004.331713] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6004.340168] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6004.346642] do_raw_spin_lock+0x177/0x190 [ 6004.355605] _raw_spin_lock+0x21/0x30 [ 6004.359311] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6004.368583] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6004.374393] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6004.381526] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6004.389800] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6004.393213] ? woken_wake_function+0x30/0x30 [ 6004.403003] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6004.409566] kthread+0x1d1/0x200 [ 6004.414890] ? set_kthread_struct+0x70/0x70 [ 6004.419319] ret_from_fork+0x1f/0x30 [ 6012.062453] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 6012.071239] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6012.115284] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6012.127710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6012.138954] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6012.140608] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6012.163148] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6012.173247] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6012.182339] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6012.184747] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6012.198329] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6012.213464] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6012.224762] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6012.240718] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6012.245245] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6012.259984] Call Trace: [ 6012.263441] [ 6012.264112] ? show_regs.cold.9+0x22/0x2f [ 6012.266706] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6012.283984] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6012.285414] ? watchdog+0x40/0x40 [ 6012.286476] ? hrtimer_interrupt+0x128/0x2c0 [ 6012.288725] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6012.291498] ? apic_timer_interrupt+0xf/0x20 [ 6012.297994] [ 6012.298722] ? native_safe_halt+0x2e/0x50 [ 6012.308373] kvm_wait+0xcd/0xe0 [ 6012.311706] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6012.329849] do_raw_spin_lock+0x177/0x190 [ 6012.335419] _raw_spin_lock+0x21/0x30 [ 6012.338489] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6012.342097] ? do_raw_spin_unlock+0x75/0x190 [ 6012.344363] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6012.348381] ? do_wait_intr+0xf0/0xf0 [ 6012.350892] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6012.354965] mdc_reint+0x2d/0x110 [mdc] [ 6012.358259] mdc_unlink+0x658/0x870 [mdc] [ 6012.361197] lmv_unlink+0x34d/0xbd0 [lmv] [ 6012.365392] ll_unlink+0x24d/0x7f0 [lustre] [ 6012.368249] vfs_unlink+0x202/0x330 [ 6012.371075] do_unlinkat+0x2fa/0x460 [ 6012.374693] __x64_sys_unlink+0x37/0x50 [ 6012.378936] do_syscall_64+0xc1/0x440 [ 6012.380198] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6012.386250] RIP: 0033:0x7f83e5f5c10b [ 6012.397292] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6012.408005] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6012.410457] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6012.413370] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6012.416391] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6012.419146] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6012.423354] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6013.924665] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3500s! [ 6013.930781] Showing busy workqueues and worker pools: [ 6013.939784] workqueue events: flags=0x0 [ 6013.945098] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6013.949722] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6013.954240] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6013.961510] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6013.965530] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6013.970792] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6013.977476] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6013.982627] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6013.988194] workqueue events_long: flags=0x0 [ 6013.991418] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6013.997427] pending: lu_objects_flush [obdclass] [ 6013.999842] workqueue events_unbound: flags=0x2 [ 6014.001651] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6014.008689] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6014.011260] workqueue events_power_efficient: flags=0x80 [ 6014.013446] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6014.021582] pending: neigh_periodic_work, do_cache_clean [ 6014.027988] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6014.031944] pending: neigh_periodic_work, check_lifetime [ 6014.038565] workqueue mm_percpu_wq: flags=0x8 [ 6014.043990] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6014.048016] pending: vmstat_update, lru_add_drain_per_cpu [ 6014.053575] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6014.058445] pending: vmstat_update [ 6014.060442] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6014.067796] pending: lru_add_drain_per_cpu BAR(48) [ 6014.071948] workqueue ipv6_addrconf: flags=0x40008 [ 6014.074555] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6014.079546] pending: addrconf_verify_work [ 6014.080958] workqueue knbd0-recv: flags=0x1a [ 6014.082897] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6014.085873] in-flight: 650:recv_work [ 6014.088086] workqueue ptlrpc_pinger: flags=0x2 [ 6014.090431] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6014.095206] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6014.098879] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3500s workers=4 idle: 36 466 29757 [ 6014.102338] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6004s workers=2 idle: 652 [ 6028.060028] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 6028.064622] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6028.086622] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6028.090983] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6028.094854] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6028.098464] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6028.100765] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6028.107595] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6028.109962] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6028.112248] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6028.114895] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6028.117525] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6028.120186] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6028.122923] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6028.125316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6028.127207] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6028.129733] Call Trace: [ 6028.130578] [ 6028.131228] ? show_regs.cold.9+0x22/0x2f [ 6028.132578] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6028.134075] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6028.135520] ? watchdog+0x40/0x40 [ 6028.136666] ? hrtimer_interrupt+0x128/0x2c0 [ 6028.138090] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6028.139900] ? apic_timer_interrupt+0xf/0x20 [ 6028.141191] [ 6028.142025] ? native_safe_halt+0x2e/0x50 [ 6028.144165] kvm_wait+0xcd/0xe0 [ 6028.145356] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6028.150146] do_raw_spin_lock+0x177/0x190 [ 6028.151994] _raw_spin_lock+0x21/0x30 [ 6028.163767] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6028.169621] process_one_work+0x2c8/0x700 [ 6028.172619] worker_thread+0x296/0x6e0 [ 6028.175202] ? rescuer_thread+0x570/0x570 [ 6028.179193] kthread+0x1d1/0x200 [ 6028.182034] ? set_kthread_struct+0x70/0x70 [ 6028.184840] ret_from_fork+0x1f/0x30 [ 6032.059028] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 6032.063202] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6032.092280] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6032.101417] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6032.109354] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6032.112357] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6032.126681] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6032.130830] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6032.135104] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6032.138272] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6032.142409] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6032.145536] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6032.148879] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6032.152631] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6032.155992] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6032.159576] Call Trace: [ 6032.160998] [ 6032.161967] ? show_regs.cold.9+0x22/0x2f [ 6032.164597] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6032.167281] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6032.170075] ? watchdog+0x40/0x40 [ 6032.172052] ? hrtimer_interrupt+0x128/0x2c0 [ 6032.174816] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6032.178032] ? apic_timer_interrupt+0xf/0x20 [ 6032.180009] [ 6032.181080] ? native_safe_halt+0x2e/0x50 [ 6032.183414] kvm_wait+0xcd/0xe0 [ 6032.185055] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6032.187251] do_raw_spin_lock+0x177/0x190 [ 6032.189096] _raw_spin_lock+0x21/0x30 [ 6032.191137] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6032.195255] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6032.198749] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6032.202390] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6032.205281] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6032.207941] ? woken_wake_function+0x30/0x30 [ 6032.209892] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6032.213797] kthread+0x1d1/0x200 [ 6032.216068] ? set_kthread_struct+0x70/0x70 [ 6032.218223] ret_from_fork+0x1f/0x30 [ 6040.056202] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 6040.060890] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6040.063075] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 6040.085061] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6040.085075] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6040.085079] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6040.085094] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6040.085099] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6040.085105] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6040.085107] RDX: 000000000000749d RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6040.085110] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6040.085112] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6040.085114] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6040.085118] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6040.085121] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6040.085128] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6040.085138] Call Trace: [ 6040.085144] [ 6040.085150] ? show_regs.cold.9+0x22/0x2f [ 6040.085160] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6040.085166] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6040.085171] ? watchdog+0x40/0x40 [ 6040.085175] ? hrtimer_interrupt+0x128/0x2c0 [ 6040.085179] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6040.085188] ? apic_timer_interrupt+0xf/0x20 [ 6040.085193] [ 6040.085196] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6040.085201] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6040.085206] do_raw_spin_lock+0x177/0x190 [ 6040.085210] _raw_spin_lock+0x21/0x30 [ 6040.085217] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6040.085699] obd_mod_free+0x46/0x140 [ptlrpc] [ 6040.085970] mdc_commit_open+0x91/0xb0 [mdc] [ 6040.086106] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6040.086411] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6040.086764] after_reply+0xe2c/0x1780 [ptlrpc] [ 6040.086963] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6040.087163] ? __next_timer_interrupt+0x160/0x160 [ 6040.087173] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6040.087549] ? do_wait_intr+0xf0/0xf0 [ 6040.087571] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6040.087895] kthread+0x1d1/0x200 [ 6040.087902] ? set_kthread_struct+0x70/0x70 [ 6040.087906] ret_from_fork+0x1f/0x30 [ 6040.201901] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6040.225319] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6040.229909] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6040.234286] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6040.236533] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6040.245179] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6040.248775] RAX: 0000000000003a91 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6040.251441] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6040.254783] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6040.257158] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6040.260289] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6040.263249] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6040.266710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6040.269116] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6040.271904] Call Trace: [ 6040.272619] [ 6040.273310] ? show_regs.cold.9+0x22/0x2f [ 6040.275002] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6040.277349] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6040.279161] ? watchdog+0x40/0x40 [ 6040.281166] ? hrtimer_interrupt+0x128/0x2c0 [ 6040.283280] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6040.286566] ? apic_timer_interrupt+0xf/0x20 [ 6040.288530] [ 6040.289288] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6040.291249] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6040.293557] do_raw_spin_lock+0x177/0x190 [ 6040.295315] _raw_spin_lock+0x21/0x30 [ 6040.297088] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6040.299893] ? do_raw_spin_unlock+0x75/0x190 [ 6040.301595] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6040.304376] ? do_wait_intr+0xf0/0xf0 [ 6040.305857] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6040.308575] mdc_reint+0x2d/0x110 [mdc] [ 6040.310050] mdc_unlink+0x658/0x870 [mdc] [ 6040.312175] lmv_unlink+0x34d/0xbd0 [lmv] [ 6040.314250] ll_unlink+0x24d/0x7f0 [lustre] [ 6040.316215] vfs_unlink+0x202/0x330 [ 6040.317928] do_unlinkat+0x2fa/0x460 [ 6040.319961] __x64_sys_unlink+0x37/0x50 [ 6040.321931] do_syscall_64+0xc1/0x440 [ 6040.323366] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6040.325466] RIP: 0033:0x7f83e5f5c10b [ 6040.326867] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6040.336301] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6040.340150] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6040.343466] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6040.346877] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6040.350505] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6040.354264] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6044.128574] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3530s! [ 6044.134534] Showing busy workqueues and worker pools: [ 6044.141981] workqueue events: flags=0x0 [ 6044.147518] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6044.154887] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6044.160383] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6044.168650] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6044.175294] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6044.182950] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6044.199991] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6044.210755] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6044.218467] workqueue events_long: flags=0x0 [ 6044.224995] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6044.233942] pending: lu_objects_flush [obdclass] [ 6044.240204] workqueue events_unbound: flags=0x2 [ 6044.244018] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6044.249838] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6044.257914] workqueue events_power_efficient: flags=0x80 [ 6044.262617] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6044.270026] pending: neigh_periodic_work, do_cache_clean [ 6044.282382] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6044.288052] pending: neigh_periodic_work, check_lifetime [ 6044.293934] workqueue mm_percpu_wq: flags=0x8 [ 6044.303569] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6044.309994] pending: vmstat_update, lru_add_drain_per_cpu [ 6044.313500] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6044.322980] pending: vmstat_update [ 6044.327079] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6044.333616] pending: lru_add_drain_per_cpu BAR(48) [ 6044.339434] workqueue ipv6_addrconf: flags=0x40008 [ 6044.341648] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6044.348968] pending: addrconf_verify_work [ 6044.353935] workqueue knbd0-recv: flags=0x1a [ 6044.357013] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6044.365134] in-flight: 650:recv_work [ 6044.368195] workqueue ptlrpc_pinger: flags=0x2 [ 6044.383168] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6044.396050] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6044.405243] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3531s workers=4 idle: 36 466 29757 [ 6044.418732] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6034s workers=2 idle: 652 [ 6056.060220] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 6056.069050] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6056.114320] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6056.120686] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6056.127747] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6056.138031] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6056.143454] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6056.155233] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6056.159902] RAX: 000000000000030e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6056.164879] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6056.168029] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6056.178692] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6056.184939] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6056.189980] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6056.195027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6056.197897] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6056.202939] Call Trace: [ 6056.205040] [ 6056.206194] ? show_regs.cold.9+0x22/0x2f [ 6056.208648] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6056.211254] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6056.214978] ? watchdog+0x40/0x40 [ 6056.217163] ? hrtimer_interrupt+0x128/0x2c0 [ 6056.221024] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6056.224536] ? apic_timer_interrupt+0xf/0x20 [ 6056.227779] [ 6056.228914] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6056.231976] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6056.236346] do_raw_spin_lock+0x177/0x190 [ 6056.243259] _raw_spin_lock+0x21/0x30 [ 6056.252477] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6056.256549] process_one_work+0x2c8/0x700 [ 6056.261026] worker_thread+0x296/0x6e0 [ 6056.264734] ? rescuer_thread+0x570/0x570 [ 6056.269333] kthread+0x1d1/0x200 [ 6056.276002] ? set_kthread_struct+0x70/0x70 [ 6056.281891] ret_from_fork+0x1f/0x30 [ 6060.066389] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 6060.080819] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6060.175777] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6060.192339] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6060.205113] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6060.207318] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6060.246450] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6060.263101] RAX: 0000000000004c2e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6060.271098] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6060.283623] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6060.299557] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6060.311678] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6060.322398] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6060.333670] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6060.335859] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6060.353704] Call Trace: [ 6060.355617] [ 6060.359283] ? show_regs.cold.9+0x22/0x2f [ 6060.362542] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6060.380466] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6060.382108] ? watchdog+0x40/0x40 [ 6060.385837] ? hrtimer_interrupt+0x128/0x2c0 [ 6060.392815] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6060.400614] ? apic_timer_interrupt+0xf/0x20 [ 6060.415925] [ 6060.416674] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6060.418609] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6060.444884] do_raw_spin_lock+0x177/0x190 [ 6060.451707] _raw_spin_lock+0x21/0x30 [ 6060.457349] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6060.466738] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6060.476620] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6060.480244] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6060.504854] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6060.519542] ? woken_wake_function+0x30/0x30 [ 6060.526270] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6060.534399] kthread+0x1d1/0x200 [ 6060.541709] ? set_kthread_struct+0x70/0x70 [ 6060.555503] ret_from_fork+0x1f/0x30 [ 6068.064077] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 6068.075908] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6068.109625] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6068.117116] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6068.124704] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6068.131228] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6068.144415] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6068.147940] RAX: 00000000000007b7 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6068.151347] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6068.154678] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6068.157997] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6068.161452] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6068.164742] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6068.168078] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6068.170849] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6068.174212] Call Trace: [ 6068.175599] [ 6068.176667] ? show_regs.cold.9+0x22/0x2f [ 6068.178582] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6068.180755] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6068.182959] ? watchdog+0x40/0x40 [ 6068.184838] ? hrtimer_interrupt+0x128/0x2c0 [ 6068.187197] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6068.188994] ? apic_timer_interrupt+0xf/0x20 [ 6068.190614] [ 6068.191667] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6068.194523] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6068.197303] do_raw_spin_lock+0x177/0x190 [ 6068.199267] _raw_spin_lock+0x21/0x30 [ 6068.200908] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6068.203653] ? do_raw_spin_unlock+0x75/0x190 [ 6068.205741] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6068.208126] ? do_wait_intr+0xf0/0xf0 [ 6068.209835] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6068.212475] mdc_reint+0x2d/0x110 [mdc] [ 6068.214415] mdc_unlink+0x658/0x870 [mdc] [ 6068.216607] lmv_unlink+0x34d/0xbd0 [lmv] [ 6068.218274] ll_unlink+0x24d/0x7f0 [lustre] [ 6068.220468] vfs_unlink+0x202/0x330 [ 6068.221951] do_unlinkat+0x2fa/0x460 [ 6068.223849] __x64_sys_unlink+0x37/0x50 [ 6068.225811] do_syscall_64+0xc1/0x440 [ 6068.227667] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6068.230393] RIP: 0033:0x7f83e5f5c10b [ 6068.231815] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6068.240696] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6068.243836] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6068.247031] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6068.250665] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6068.254072] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6068.257152] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6072.056010] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 6072.062918] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6072.084792] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6072.088696] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6072.091872] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6072.093889] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6072.101844] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6072.105513] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6072.108108] RDX: 0000000000003695 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6072.110720] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6072.113481] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6072.116647] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6072.120479] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6072.127605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6072.132656] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6072.136572] Call Trace: [ 6072.137520] [ 6072.138588] ? show_regs.cold.9+0x22/0x2f [ 6072.140583] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6072.142963] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6072.145151] ? watchdog+0x40/0x40 [ 6072.146527] ? hrtimer_interrupt+0x128/0x2c0 [ 6072.149122] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6072.151184] ? apic_timer_interrupt+0xf/0x20 [ 6072.153472] [ 6072.154482] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6072.157278] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6072.159904] do_raw_spin_lock+0x177/0x190 [ 6072.161702] _raw_spin_lock+0x21/0x30 [ 6072.163474] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6072.166189] obd_mod_free+0x46/0x140 [ptlrpc] [ 6072.168491] mdc_commit_open+0x91/0xb0 [mdc] [ 6072.170453] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6072.175908] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6072.182030] after_reply+0xe2c/0x1780 [ptlrpc] [ 6072.184425] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6072.186928] ? __next_timer_interrupt+0x160/0x160 [ 6072.189216] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6072.190926] ? do_wait_intr+0xf0/0xf0 [ 6072.192715] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6072.194773] kthread+0x1d1/0x200 [ 6072.196203] ? set_kthread_struct+0x70/0x70 [ 6072.198012] ret_from_fork+0x1f/0x30 [ 6074.850049] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3561s! [ 6074.855200] Showing busy workqueues and worker pools: [ 6074.857165] workqueue events: flags=0x0 [ 6074.861285] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6074.864471] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6074.868503] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6074.873093] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6074.877298] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6074.880677] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6074.884278] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6074.886795] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6074.890855] workqueue events_long: flags=0x0 [ 6074.892812] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6074.895988] pending: lu_objects_flush [obdclass] [ 6074.898484] workqueue events_unbound: flags=0x2 [ 6074.900205] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6074.903135] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6074.904909] workqueue events_power_efficient: flags=0x80 [ 6074.906479] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6074.909413] pending: neigh_periodic_work, do_cache_clean [ 6074.911777] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6074.914637] pending: neigh_periodic_work, check_lifetime [ 6074.916968] workqueue mm_percpu_wq: flags=0x8 [ 6074.918777] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6074.921644] pending: vmstat_update, lru_add_drain_per_cpu [ 6074.924259] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6074.927103] pending: vmstat_update [ 6074.928602] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6074.931445] pending: lru_add_drain_per_cpu BAR(48) [ 6074.933520] workqueue ipv6_addrconf: flags=0x40008 [ 6074.935457] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6074.938045] pending: addrconf_verify_work [ 6074.939951] workqueue knbd0-recv: flags=0x1a [ 6074.941736] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6074.944307] in-flight: 650:recv_work [ 6074.946098] workqueue ptlrpc_pinger: flags=0x2 [ 6074.947864] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6074.950328] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6074.952740] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3561s workers=4 idle: 36 466 29757 [ 6074.955977] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6065s workers=2 idle: 652 [ 6084.060097] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 6084.064274] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6084.088764] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6084.095800] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6084.101046] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6084.104518] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6084.106421] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6084.114569] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6084.117954] RAX: 0000000000003705 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6084.120666] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6084.123596] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6084.126787] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6084.129689] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6084.132424] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6084.138550] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6084.140793] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6084.143811] Call Trace: [ 6084.145303] [ 6084.146268] ? show_regs.cold.9+0x22/0x2f [ 6084.147802] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6084.149977] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6084.152735] ? watchdog+0x40/0x40 [ 6084.153728] ? hrtimer_interrupt+0x128/0x2c0 [ 6084.155238] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6084.156473] ? apic_timer_interrupt+0xf/0x20 [ 6084.157545] [ 6084.158203] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6084.159796] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6084.161717] do_raw_spin_lock+0x177/0x190 [ 6084.163596] _raw_spin_lock+0x21/0x30 [ 6084.164696] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6084.166674] process_one_work+0x2c8/0x700 [ 6084.167730] worker_thread+0x296/0x6e0 [ 6084.168770] ? rescuer_thread+0x570/0x570 [ 6084.170620] kthread+0x1d1/0x200 [ 6084.171677] ? set_kthread_struct+0x70/0x70 [ 6084.173307] ret_from_fork+0x1f/0x30 [ 6088.060378] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 6088.066927] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6088.094547] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6088.101184] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6088.105535] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6088.107752] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6088.117243] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6088.120872] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6088.124402] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6088.127357] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6088.132027] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6088.135924] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6088.139545] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6088.143250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6088.145735] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6088.149423] Call Trace: [ 6088.150760] [ 6088.151745] ? show_regs.cold.9+0x22/0x2f [ 6088.153930] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6088.156551] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6088.159225] ? watchdog+0x40/0x40 [ 6088.160916] ? hrtimer_interrupt+0x128/0x2c0 [ 6088.162989] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6088.165083] ? apic_timer_interrupt+0xf/0x20 [ 6088.166936] [ 6088.167832] ? native_safe_halt+0x2e/0x50 [ 6088.169716] kvm_wait+0xcd/0xe0 [ 6088.171323] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6088.173755] do_raw_spin_lock+0x177/0x190 [ 6088.175448] _raw_spin_lock+0x21/0x30 [ 6088.177189] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6088.181373] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6088.184196] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6088.187062] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6088.189801] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6088.192799] ? woken_wake_function+0x30/0x30 [ 6088.194863] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6088.198784] kthread+0x1d1/0x200 [ 6088.200326] ? set_kthread_struct+0x70/0x70 [ 6088.202823] ret_from_fork+0x1f/0x30 [ 6096.062021] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6096.066544] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6096.094987] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6096.106899] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6096.113806] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6096.115784] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6096.124863] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6096.128006] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6096.130893] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6096.133233] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6096.135406] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6096.137896] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6096.140968] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6096.144224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6096.146642] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6096.154676] Call Trace: [ 6096.155695] [ 6096.156427] ? show_regs.cold.9+0x22/0x2f [ 6096.163705] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6096.165494] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6096.167424] ? watchdog+0x40/0x40 [ 6096.168649] ? hrtimer_interrupt+0x128/0x2c0 [ 6096.176864] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6096.179518] ? apic_timer_interrupt+0xf/0x20 [ 6096.182625] [ 6096.184283] ? native_safe_halt+0x2e/0x50 [ 6096.186008] kvm_wait+0xcd/0xe0 [ 6096.187578] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6096.190376] do_raw_spin_lock+0x177/0x190 [ 6096.192288] _raw_spin_lock+0x21/0x30 [ 6096.193894] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6096.202646] ? do_raw_spin_unlock+0x75/0x190 [ 6096.204290] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6096.210779] ? do_wait_intr+0xf0/0xf0 [ 6096.212297] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6096.215997] mdc_reint+0x2d/0x110 [mdc] [ 6096.216989] mdc_unlink+0x658/0x870 [mdc] [ 6096.218022] lmv_unlink+0x34d/0xbd0 [lmv] [ 6096.228326] ll_unlink+0x24d/0x7f0 [lustre] [ 6096.230064] vfs_unlink+0x202/0x330 [ 6096.231757] do_unlinkat+0x2fa/0x460 [ 6096.234419] __x64_sys_unlink+0x37/0x50 [ 6096.239763] do_syscall_64+0xc1/0x440 [ 6096.241712] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6096.244517] RIP: 0033:0x7f83e5f5c10b [ 6096.247982] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6096.263718] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6096.269363] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6096.278423] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6096.284823] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6096.292959] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6096.300686] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6100.056220] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 6100.065534] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6100.115843] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6100.131997] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6100.155248] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6100.159798] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6100.172375] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6100.175736] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6100.181454] RDX: 00000000000058dd RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6100.185056] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6100.188845] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6100.192247] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6100.195520] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6100.199195] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6100.201609] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6100.205384] Call Trace: [ 6100.206424] [ 6100.207478] ? show_regs.cold.9+0x22/0x2f [ 6100.209298] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6100.212608] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6100.221223] ? watchdog+0x40/0x40 [ 6100.226188] ? hrtimer_interrupt+0x128/0x2c0 [ 6100.227642] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6100.238454] ? apic_timer_interrupt+0xf/0x20 [ 6100.246866] [ 6100.249148] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6100.256766] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6100.260883] do_raw_spin_lock+0x177/0x190 [ 6100.262294] _raw_spin_lock+0x21/0x30 [ 6100.278060] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6100.283368] obd_mod_free+0x46/0x140 [ptlrpc] [ 6100.291816] mdc_commit_open+0x91/0xb0 [mdc] [ 6100.297609] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6100.307557] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6100.311695] after_reply+0xe2c/0x1780 [ptlrpc] [ 6100.316703] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6100.323634] ? __next_timer_interrupt+0x160/0x160 [ 6100.331936] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6100.340652] ? do_wait_intr+0xf0/0xf0 [ 6100.343406] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6100.350515] kthread+0x1d1/0x200 [ 6100.352584] ? set_kthread_struct+0x70/0x70 [ 6100.353930] ret_from_fork+0x1f/0x30 [ 6105.060188] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3591s! [ 6105.067153] Showing busy workqueues and worker pools: [ 6105.068947] workqueue events: flags=0x0 [ 6105.073147] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6105.078104] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6105.084619] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6105.092476] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6105.094701] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6105.106553] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6105.118150] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6105.126406] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6105.144557] workqueue events_long: flags=0x0 [ 6105.152610] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6105.165171] pending: lu_objects_flush [obdclass] [ 6105.174366] workqueue events_unbound: flags=0x2 [ 6105.179458] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6105.198213] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6105.209926] workqueue events_power_efficient: flags=0x80 [ 6105.217011] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6105.230217] pending: neigh_periodic_work, do_cache_clean [ 6105.232173] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6105.247280] pending: neigh_periodic_work, check_lifetime [ 6105.256889] workqueue mm_percpu_wq: flags=0x8 [ 6105.261259] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6105.269982] pending: vmstat_update, lru_add_drain_per_cpu [ 6105.274649] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6105.276813] pending: vmstat_update [ 6105.288457] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6105.299552] pending: lru_add_drain_per_cpu BAR(48) [ 6105.304181] workqueue ipv6_addrconf: flags=0x40008 [ 6105.311637] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6105.317694] pending: addrconf_verify_work [ 6105.320262] workqueue knbd0-recv: flags=0x1a [ 6105.326195] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6105.330405] in-flight: 650:recv_work [ 6105.333782] workqueue ptlrpc_pinger: flags=0x2 [ 6105.336145] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6105.347747] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6105.353775] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3592s workers=4 idle: 36 466 29757 [ 6105.356583] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6095s workers=2 idle: 652 [ 6112.060028] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 6112.066700] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6112.089374] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6112.103816] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6112.107104] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6112.111910] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6112.113983] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6112.147174] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6112.152165] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6112.154104] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6112.157509] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6112.161345] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6112.164804] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6112.168023] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6112.171522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6112.174445] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6112.177429] Call Trace: [ 6112.178506] [ 6112.179213] ? show_regs.cold.9+0x22/0x2f [ 6112.180633] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6112.183157] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6112.184228] ? watchdog+0x40/0x40 [ 6112.185157] ? hrtimer_interrupt+0x128/0x2c0 [ 6112.186369] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6112.187540] ? apic_timer_interrupt+0xf/0x20 [ 6112.188580] [ 6112.189245] ? native_safe_halt+0x2e/0x50 [ 6112.190207] kvm_wait+0xcd/0xe0 [ 6112.191411] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6112.192831] do_raw_spin_lock+0x177/0x190 [ 6112.193785] _raw_spin_lock+0x21/0x30 [ 6112.194733] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6112.196292] process_one_work+0x2c8/0x700 [ 6112.197463] worker_thread+0x296/0x6e0 [ 6112.198344] ? rescuer_thread+0x570/0x570 [ 6112.199369] kthread+0x1d1/0x200 [ 6112.200251] ? set_kthread_struct+0x70/0x70 [ 6112.201348] ret_from_fork+0x1f/0x30 [ 6116.059007] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 6116.061687] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6116.091983] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6116.096538] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6116.099592] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6116.103363] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6116.127046] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6116.132423] RAX: 0000000000007d93 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6116.139383] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6116.147606] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6116.153171] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6116.167154] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6116.181857] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6116.192157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6116.202573] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6116.210439] Call Trace: [ 6116.215428] [ 6116.217125] ? show_regs.cold.9+0x22/0x2f [ 6116.221089] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6116.224999] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6116.228254] ? watchdog+0x40/0x40 [ 6116.230118] ? hrtimer_interrupt+0x128/0x2c0 [ 6116.232342] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6116.238257] ? apic_timer_interrupt+0xf/0x20 [ 6116.241685] [ 6116.244958] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6116.251966] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6116.262125] do_raw_spin_lock+0x177/0x190 [ 6116.267929] _raw_spin_lock+0x21/0x30 [ 6116.277149] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6116.284241] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6116.289089] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6116.293706] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6116.303717] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6116.311056] ? woken_wake_function+0x30/0x30 [ 6116.316925] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6116.321748] kthread+0x1d1/0x200 [ 6116.328579] ? set_kthread_struct+0x70/0x70 [ 6116.334671] ret_from_fork+0x1f/0x30 [ 6124.062030] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6124.065809] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6124.085188] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6124.089395] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6124.093003] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6124.094631] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6124.101335] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6124.104167] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6124.106946] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6124.110575] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6124.113592] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6124.117717] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6124.121050] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6124.124822] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6124.127451] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6124.130313] Call Trace: [ 6124.130892] [ 6124.131363] ? show_regs.cold.9+0x22/0x2f [ 6124.132605] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6124.133693] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6124.135719] ? watchdog+0x40/0x40 [ 6124.137059] ? hrtimer_interrupt+0x128/0x2c0 [ 6124.138372] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6124.139599] ? apic_timer_interrupt+0xf/0x20 [ 6124.140663] [ 6124.142122] ? native_safe_halt+0x2e/0x50 [ 6124.144140] kvm_wait+0xcd/0xe0 [ 6124.145528] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6124.148319] do_raw_spin_lock+0x177/0x190 [ 6124.149363] _raw_spin_lock+0x21/0x30 [ 6124.150557] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6124.152198] ? do_raw_spin_unlock+0x75/0x190 [ 6124.153286] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6124.155793] ? do_wait_intr+0xf0/0xf0 [ 6124.157229] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6124.159646] mdc_reint+0x2d/0x110 [mdc] [ 6124.161289] mdc_unlink+0x658/0x870 [mdc] [ 6124.163121] lmv_unlink+0x34d/0xbd0 [lmv] [ 6124.165519] ll_unlink+0x24d/0x7f0 [lustre] [ 6124.167234] vfs_unlink+0x202/0x330 [ 6124.168289] do_unlinkat+0x2fa/0x460 [ 6124.169277] __x64_sys_unlink+0x37/0x50 [ 6124.171885] do_syscall_64+0xc1/0x440 [ 6124.173100] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6124.175173] RIP: 0033:0x7f83e5f5c10b [ 6124.177576] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6124.188497] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6124.191917] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6124.196322] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6124.200295] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6124.204410] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6124.207973] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6132.056048] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 6132.058638] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6132.102005] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6132.114278] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6132.123340] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6132.127335] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6132.149652] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6132.153172] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6132.159300] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6132.166828] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6132.170968] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6132.180437] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6132.201957] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6132.204829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6132.211446] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6132.215411] Call Trace: [ 6132.217107] [ 6132.218867] ? show_regs.cold.9+0x22/0x2f [ 6132.222962] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6132.226109] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6132.230375] ? watchdog+0x40/0x40 [ 6132.232657] ? hrtimer_interrupt+0x128/0x2c0 [ 6132.235317] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6132.239218] ? apic_timer_interrupt+0xf/0x20 [ 6132.245022] [ 6132.246386] ? native_safe_halt+0x2e/0x50 [ 6132.249524] kvm_wait+0xcd/0xe0 [ 6132.252015] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6132.254475] do_raw_spin_lock+0x177/0x190 [ 6132.257004] _raw_spin_lock+0x21/0x30 [ 6132.259322] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6132.263175] obd_mod_free+0x46/0x140 [ptlrpc] [ 6132.267194] mdc_commit_open+0x91/0xb0 [mdc] [ 6132.276297] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6132.284071] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6132.290314] after_reply+0xe2c/0x1780 [ptlrpc] [ 6132.298089] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6132.307800] ? __next_timer_interrupt+0x160/0x160 [ 6132.309375] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6132.311645] ? do_wait_intr+0xf0/0xf0 [ 6132.313600] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6132.316054] kthread+0x1d1/0x200 [ 6132.317373] ? set_kthread_struct+0x70/0x70 [ 6132.319378] ret_from_fork+0x1f/0x30 [ 6135.779976] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3622s! [ 6135.792086] Showing busy workqueues and worker pools: [ 6135.795405] workqueue events: flags=0x0 [ 6135.797265] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6135.802069] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6135.807418] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6135.814745] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6135.820348] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6135.823937] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6135.828043] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6135.830554] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6135.834060] workqueue events_long: flags=0x0 [ 6135.835465] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6135.838750] pending: lu_objects_flush [obdclass] [ 6135.842539] workqueue events_unbound: flags=0x2 [ 6135.844859] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6135.848111] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6135.853733] workqueue events_power_efficient: flags=0x80 [ 6135.857266] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6135.861606] pending: neigh_periodic_work, do_cache_clean [ 6135.864942] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6135.867813] pending: neigh_periodic_work, check_lifetime [ 6135.870045] workqueue mm_percpu_wq: flags=0x8 [ 6135.872018] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6135.875028] pending: vmstat_update, lru_add_drain_per_cpu [ 6135.877183] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6135.880063] pending: vmstat_update [ 6135.881859] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6135.884911] pending: lru_add_drain_per_cpu BAR(48) [ 6135.887160] workqueue ipv6_addrconf: flags=0x40008 [ 6135.889136] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6135.892366] pending: addrconf_verify_work [ 6135.894344] workqueue knbd0-recv: flags=0x1a [ 6135.896011] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6135.899388] in-flight: 650:recv_work [ 6135.900850] workqueue ptlrpc_pinger: flags=0x2 [ 6135.902782] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6135.905492] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6135.908445] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3622s workers=4 idle: 36 466 29757 [ 6135.911738] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6125s workers=2 idle: 652 [ 6140.063531] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6140.066992] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6140.091404] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6140.096866] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6140.100297] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6140.103191] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6140.106078] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6140.114903] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6140.120382] RAX: 0000000000002989 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6140.123994] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6140.127562] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6140.131101] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6140.137624] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6140.141749] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6140.145748] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6140.148826] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6140.155860] Call Trace: [ 6140.157925] [ 6140.159411] ? show_regs.cold.9+0x22/0x2f [ 6140.160892] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6140.163833] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6140.166063] ? watchdog+0x40/0x40 [ 6140.167466] ? hrtimer_interrupt+0x128/0x2c0 [ 6140.169289] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6140.171697] ? apic_timer_interrupt+0xf/0x20 [ 6140.173573] [ 6140.174322] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6140.177003] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6140.179762] do_raw_spin_lock+0x177/0x190 [ 6140.181516] _raw_spin_lock+0x21/0x30 [ 6140.183346] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6140.186419] process_one_work+0x2c8/0x700 [ 6140.188060] worker_thread+0x296/0x6e0 [ 6140.189837] ? rescuer_thread+0x570/0x570 [ 6140.191626] kthread+0x1d1/0x200 [ 6140.192934] ? set_kthread_struct+0x70/0x70 [ 6140.194693] ret_from_fork+0x1f/0x30 [ 6144.059473] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 6144.075335] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6144.130997] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6144.146518] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6144.162611] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6144.166334] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6144.194746] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6144.202904] RAX: 0000000000007c91 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6144.209893] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6144.212269] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6144.227454] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6144.230617] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6144.239057] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6144.252825] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6144.260695] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6144.268122] Call Trace: [ 6144.271262] [ 6144.275370] ? show_regs.cold.9+0x22/0x2f [ 6144.279709] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6144.284760] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6144.287546] ? watchdog+0x40/0x40 [ 6144.289871] ? hrtimer_interrupt+0x128/0x2c0 [ 6144.292745] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6144.295329] ? apic_timer_interrupt+0xf/0x20 [ 6144.297866] [ 6144.299072] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6144.301206] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6144.304257] do_raw_spin_lock+0x177/0x190 [ 6144.306284] _raw_spin_lock+0x21/0x30 [ 6144.307867] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6144.310949] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6144.314760] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6144.318321] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6144.321818] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6144.325805] ? woken_wake_function+0x30/0x30 [ 6144.334528] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6144.346107] kthread+0x1d1/0x200 [ 6144.352575] ? set_kthread_struct+0x70/0x70 [ 6144.357976] ret_from_fork+0x1f/0x30 [ 6152.067804] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6152.072986] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6152.125631] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6152.134841] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6152.140003] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6152.146359] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6152.161952] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6152.166366] RAX: 0000000000006598 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6152.169409] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6152.180833] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6152.190125] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6152.198466] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6152.205749] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6152.212125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6152.218650] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6152.226514] Call Trace: [ 6152.227904] [ 6152.230646] ? show_regs.cold.9+0x22/0x2f [ 6152.232889] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6152.239165] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6152.244437] ? watchdog+0x40/0x40 [ 6152.247394] ? hrtimer_interrupt+0x128/0x2c0 [ 6152.249523] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6152.251858] ? apic_timer_interrupt+0xf/0x20 [ 6152.258227] [ 6152.260108] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6152.265694] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6152.270151] do_raw_spin_lock+0x177/0x190 [ 6152.273676] _raw_spin_lock+0x21/0x30 [ 6152.275006] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6152.286838] ? do_raw_spin_unlock+0x75/0x190 [ 6152.291041] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6152.296006] ? do_wait_intr+0xf0/0xf0 [ 6152.297437] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6152.302931] mdc_reint+0x2d/0x110 [mdc] [ 6152.304970] mdc_unlink+0x658/0x870 [mdc] [ 6152.307014] lmv_unlink+0x34d/0xbd0 [lmv] [ 6152.312370] ll_unlink+0x24d/0x7f0 [lustre] [ 6152.316517] vfs_unlink+0x202/0x330 [ 6152.320090] do_unlinkat+0x2fa/0x460 [ 6152.322040] __x64_sys_unlink+0x37/0x50 [ 6152.324159] do_syscall_64+0xc1/0x440 [ 6152.325752] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6152.329774] RIP: 0033:0x7f83e5f5c10b [ 6152.332224] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6152.350859] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6152.356626] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6152.360574] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6152.366954] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6152.372438] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6152.377461] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6160.060555] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 6160.075283] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6160.122171] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6160.137366] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6160.146601] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6160.149900] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6160.172520] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6160.181519] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6160.187151] RDX: 00000000000039f9 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6160.196688] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6160.201965] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6160.211799] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6160.214056] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6160.231866] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6160.236551] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6160.246211] Call Trace: [ 6160.248882] [ 6160.251077] ? show_regs.cold.9+0x22/0x2f [ 6160.254482] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6160.256902] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6160.264722] ? watchdog+0x40/0x40 [ 6160.269780] ? hrtimer_interrupt+0x128/0x2c0 [ 6160.272737] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6160.278065] ? apic_timer_interrupt+0xf/0x20 [ 6160.281720] [ 6160.283710] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6160.293080] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6160.297719] do_raw_spin_lock+0x177/0x190 [ 6160.299740] _raw_spin_lock+0x21/0x30 [ 6160.305609] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6160.310668] obd_mod_free+0x46/0x140 [ptlrpc] [ 6160.313424] mdc_commit_open+0x91/0xb0 [mdc] [ 6160.314889] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6160.317845] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6160.328670] after_reply+0xe2c/0x1780 [ptlrpc] [ 6160.331180] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6160.334083] ? __next_timer_interrupt+0x160/0x160 [ 6160.342842] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6160.346335] ? do_wait_intr+0xf0/0xf0 [ 6160.349652] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6160.352730] kthread+0x1d1/0x200 [ 6160.356552] ? set_kthread_struct+0x70/0x70 [ 6160.363150] ret_from_fork+0x1f/0x30 [ 6165.991815] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3652s! [ 6166.003387] Showing busy workqueues and worker pools: [ 6166.011166] workqueue events: flags=0x0 [ 6166.015775] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6166.020683] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6166.025978] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6166.032030] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6166.034849] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6166.044846] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6166.050813] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6166.060890] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6166.071097] workqueue events_long: flags=0x0 [ 6166.080394] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6166.083847] pending: lu_objects_flush [obdclass] [ 6166.087652] workqueue events_unbound: flags=0x2 [ 6166.093759] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6166.097952] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6166.103148] workqueue events_power_efficient: flags=0x80 [ 6166.107812] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6166.112610] pending: neigh_periodic_work, do_cache_clean [ 6166.118397] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6166.123109] pending: neigh_periodic_work, check_lifetime [ 6166.130601] workqueue mm_percpu_wq: flags=0x8 [ 6166.132697] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6166.135400] pending: vmstat_update, lru_add_drain_per_cpu [ 6166.137516] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6166.142613] pending: vmstat_update [ 6166.143878] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6166.156838] pending: lru_add_drain_per_cpu BAR(48) [ 6166.158093] workqueue ipv6_addrconf: flags=0x40008 [ 6166.159305] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6166.162490] pending: addrconf_verify_work [ 6166.164320] workqueue knbd0-recv: flags=0x1a [ 6166.169103] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6166.170785] in-flight: 650:recv_work [ 6166.176725] workqueue ptlrpc_pinger: flags=0x2 [ 6166.177835] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6166.180792] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6166.182685] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3652s workers=4 idle: 36 466 29757 [ 6166.188961] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6156s workers=2 idle: 652 [ 6168.063983] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6168.069268] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6168.103267] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6168.110784] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6168.122546] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6168.126429] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6168.129661] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6168.142746] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6168.146582] RAX: 0000000000007e8e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6168.149932] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6168.153847] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6168.159150] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6168.163831] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6168.166786] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6168.171466] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6168.178827] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6168.186786] Call Trace: [ 6168.188412] [ 6168.191463] ? show_regs.cold.9+0x22/0x2f [ 6168.194627] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6168.196874] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6168.198961] ? watchdog+0x40/0x40 [ 6168.201693] ? hrtimer_interrupt+0x128/0x2c0 [ 6168.204500] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6168.207257] ? apic_timer_interrupt+0xf/0x20 [ 6168.208973] [ 6168.209929] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6168.215253] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6168.219995] do_raw_spin_lock+0x177/0x190 [ 6168.222441] _raw_spin_lock+0x21/0x30 [ 6168.224853] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6168.227762] process_one_work+0x2c8/0x700 [ 6168.231704] worker_thread+0x296/0x6e0 [ 6168.233462] ? rescuer_thread+0x570/0x570 [ 6168.235195] kthread+0x1d1/0x200 [ 6168.236745] ? set_kthread_struct+0x70/0x70 [ 6168.238713] ret_from_fork+0x1f/0x30 [ 6172.059066] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 6172.065809] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6172.107336] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6172.116630] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6172.130897] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6172.139615] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6172.155788] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6172.162707] RAX: 0000000000002135 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6172.169652] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6172.176902] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6172.184054] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6172.190269] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6172.195110] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6172.205099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6172.211701] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6172.216093] Call Trace: [ 6172.217620] [ 6172.220008] ? show_regs.cold.9+0x22/0x2f [ 6172.224768] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6172.229789] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6172.232151] ? watchdog+0x40/0x40 [ 6172.234147] ? hrtimer_interrupt+0x128/0x2c0 [ 6172.236254] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6172.241674] ? apic_timer_interrupt+0xf/0x20 [ 6172.243134] [ 6172.244359] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6172.246673] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6172.254672] do_raw_spin_lock+0x177/0x190 [ 6172.262631] _raw_spin_lock+0x21/0x30 [ 6172.264508] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6172.268133] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6172.273098] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6172.278251] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6172.283655] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6172.286662] ? woken_wake_function+0x30/0x30 [ 6172.287912] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6172.293670] kthread+0x1d1/0x200 [ 6172.295609] ? set_kthread_struct+0x70/0x70 [ 6172.299456] ret_from_fork+0x1f/0x30 [ 6180.066994] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6180.073893] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6180.109848] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6180.116341] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6180.120382] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6180.124800] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6180.134178] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6180.139015] RAX: 0000000000007698 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6180.142853] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6180.150011] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6180.153756] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6180.158331] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6180.161508] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6180.166092] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6180.169243] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6180.173102] Call Trace: [ 6180.174650] [ 6180.175811] ? show_regs.cold.9+0x22/0x2f [ 6180.178636] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6180.182304] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6180.186681] ? watchdog+0x40/0x40 [ 6180.188591] ? hrtimer_interrupt+0x128/0x2c0 [ 6180.190563] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6180.192500] ? apic_timer_interrupt+0xf/0x20 [ 6180.197489] [ 6180.198848] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6180.201592] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6180.203564] do_raw_spin_lock+0x177/0x190 [ 6180.205578] _raw_spin_lock+0x21/0x30 [ 6180.207436] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6180.210138] ? do_raw_spin_unlock+0x75/0x190 [ 6180.211829] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6180.215330] ? do_wait_intr+0xf0/0xf0 [ 6180.217313] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6180.219814] mdc_reint+0x2d/0x110 [mdc] [ 6180.222071] mdc_unlink+0x658/0x870 [mdc] [ 6180.224322] lmv_unlink+0x34d/0xbd0 [lmv] [ 6180.227345] ll_unlink+0x24d/0x7f0 [lustre] [ 6180.230534] vfs_unlink+0x202/0x330 [ 6180.233579] do_unlinkat+0x2fa/0x460 [ 6180.237770] __x64_sys_unlink+0x37/0x50 [ 6180.242479] do_syscall_64+0xc1/0x440 [ 6180.245869] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6180.250481] RIP: 0033:0x7f83e5f5c10b [ 6180.252355] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6180.264638] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6180.267558] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6180.270958] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6180.275170] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6180.279620] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6180.283017] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6181.056132] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 6181.059457] rcu: 0-....: (2662189 ticks this GP) idle=30a/1/0x4000000000000000 softirq=128519/128519 fqs=126395 [ 6181.064459] (detected by 1, t=4200097 jiffies, g=262441, q=58418) [ 6181.066805] Sending NMI from CPU 1 to CPUs 0: [ 6181.074498] NMI backtrace for cpu 0 [ 6181.074504] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6181.074507] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6181.074508] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6181.074510] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6181.074512] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 [ 6181.074514] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6181.074515] RDX: 000000000000558d RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6181.074517] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6181.074518] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6181.074519] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6181.074521] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6181.074522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6181.074523] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6181.074525] Call Trace: [ 6181.074526] [ 6181.074527] ? show_regs.cold.9+0x22/0x2f [ 6181.074528] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6181.074529] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6181.074530] ? nmi_handle+0xbd/0x210 [ 6181.074531] ? default_do_nmi+0x4b/0x1b0 [ 6181.074532] ? do_nmi+0x280/0x380 [ 6181.074533] ? end_repeat_nmi+0x16/0x1f [ 6181.074534] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6181.074535] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6181.074536] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6181.074537] [ 6181.074538] do_raw_spin_lock+0x177/0x190 [ 6181.074539] _raw_spin_lock+0x21/0x30 [ 6181.074540] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6181.074541] obd_mod_free+0x46/0x140 [ptlrpc] [ 6181.074542] mdc_commit_open+0x91/0xb0 [mdc] [ 6181.074543] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6181.074544] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6181.074545] after_reply+0xe2c/0x1780 [ptlrpc] [ 6181.074546] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6181.074547] ? __next_timer_interrupt+0x160/0x160 [ 6181.074549] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6181.074550] ? do_wait_intr+0xf0/0xf0 [ 6181.074551] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6181.074552] kthread+0x1d1/0x200 [ 6181.074553] ? set_kthread_struct+0x70/0x70 [ 6181.074554] ret_from_fork+0x1f/0x30 [ 6181.075000] rcu: rcu_sched kthread starved for 3669284 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 6181.252297] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6181.255470] rcu: RCU grace-period kthread stack dump: [ 6181.258508] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6181.264018] Call Trace: [ 6181.265662] __schedule+0x351/0xcb0 [ 6181.267795] schedule+0xc0/0x180 [ 6181.269777] schedule_timeout+0xb4/0x190 [ 6181.272376] ? do_raw_spin_unlock+0x75/0x190 [ 6181.275341] ? __next_timer_interrupt+0x160/0x160 [ 6181.279278] ? prepare_to_swait_event+0xf3/0x140 [ 6181.282944] rcu_gp_kthread+0x8dd/0x15b0 [ 6181.286400] ? force_qs_rnp+0x2c0/0x2c0 [ 6181.290299] kthread+0x1d1/0x200 [ 6181.291687] ? set_kthread_struct+0x70/0x70 [ 6181.293604] ret_from_fork+0x1f/0x30 [ 6181.295312] rcu: Stack dump where RCU GP kthread last ran: [ 6181.298291] NMI backtrace for cpu 1 [ 6181.300199] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6181.310302] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6181.316451] Call Trace: [ 6181.317662] [ 6181.318809] ? dump_stack+0xbb/0x10e [ 6181.320409] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6181.323084] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6181.325626] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6181.329122] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6181.332806] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6181.335366] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 6181.336964] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [ 6181.338965] ? update_process_times+0x6d/0xc0 [ 6181.340468] ? tick_sched_handle.isra.18+0x27/0x90 [ 6181.342588] ? tick_sched_timer+0x53/0xb0 [ 6181.343995] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6181.345728] ? tick_sched_handle.isra.18+0x90/0x90 [ 6181.347316] ? hrtimer_interrupt+0x128/0x2c0 [ 6181.348570] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6181.350111] ? apic_timer_interrupt+0xf/0x20 [ 6181.352610] [ 6181.353594] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6181.356268] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6181.358781] ? do_raw_spin_lock+0x177/0x190 [ 6181.360611] ? _raw_spin_lock+0x21/0x30 [ 6181.362412] ? ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6181.369073] ? ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6181.373418] ? ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6181.375944] ? ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6181.377848] ? ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6181.384600] ? woken_wake_function+0x30/0x30 [ 6181.388508] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6181.392515] ? kthread+0x1d1/0x200 [ 6181.393786] ? set_kthread_struct+0x70/0x70 [ 6181.400247] ? ret_from_fork+0x1f/0x30 [ 6196.060034] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6196.062356] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6196.097909] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6196.102697] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6196.106734] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6196.110162] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6196.113027] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6196.122632] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6196.126031] RAX: 0000000000007385 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6196.129349] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6196.132690] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6196.136909] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6196.141993] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6196.146631] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6196.150384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6196.155523] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6196.159927] Call Trace: [ 6196.161433] [ 6196.162442] ? show_regs.cold.9+0x22/0x2f [ 6196.164449] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6196.166795] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6196.169446] ? watchdog+0x40/0x40 [ 6196.171464] ? hrtimer_interrupt+0x128/0x2c0 [ 6196.173781] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6196.176676] ? apic_timer_interrupt+0xf/0x20 [ 6196.179384] [ 6196.180511] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6196.183438] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6196.186226] do_raw_spin_lock+0x177/0x190 [ 6196.188913] _raw_spin_lock+0x21/0x30 [ 6196.190746] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6196.192285] BUG: workqueue lockup - pool [ 6196.194020] process_one_work+0x2c8/0x700 [ 6196.200808] cpus=0-3 flags=0x4 nice=0 stuck for 3683s! [ 6196.203665] worker_thread+0x296/0x6e0 [ 6196.211410] Showing busy workqueues and worker pools: [ 6196.214454] ? rescuer_thread+0x570/0x570 [ 6196.214473] kthread+0x1d1/0x200 [ 6196.214477] ? set_kthread_struct+0x70/0x70 [ 6196.214480] ret_from_fork+0x1f/0x30 [ 6196.229372] workqueue events: flags=0x0 [ 6196.231195] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6196.235585] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6196.244532] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6196.254337] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6196.269917] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6196.272282] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6196.281010] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6196.286453] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6196.295505] workqueue events_long: flags=0x0 [ 6196.299305] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6196.308949] pending: lu_objects_flush [obdclass] [ 6196.310949] workqueue events_unbound: flags=0x2 [ 6196.316607] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6196.320645] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6196.325350] workqueue events_power_efficient: flags=0x80 [ 6196.336341] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6196.340421] pending: neigh_periodic_work, do_cache_clean [ 6196.342351] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6196.347594] pending: neigh_periodic_work, check_lifetime [ 6196.352144] workqueue mm_percpu_wq: flags=0x8 [ 6196.359619] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6196.366154] pending: vmstat_update, lru_add_drain_per_cpu [ 6196.370358] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6196.373856] pending: vmstat_update [ 6196.376420] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6196.382553] pending: lru_add_drain_per_cpu BAR(48) [ 6196.385640] workqueue ipv6_addrconf: flags=0x40008 [ 6196.388462] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6196.391682] pending: addrconf_verify_work [ 6196.394228] workqueue knbd0-recv: flags=0x1a [ 6196.396922] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6196.400370] in-flight: 650:recv_work [ 6196.402323] workqueue ptlrpc_pinger: flags=0x2 [ 6196.404623] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6196.407463] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6196.410899] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3683s workers=4 idle: 36 466 29757 [ 6196.415569] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6186s workers=2 idle: 652 [ 6208.059053] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 6208.062227] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6208.063755] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6208.063835] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6208.063842] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6208.063845] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6208.063859] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6208.063864] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6208.063868] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6208.063871] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6208.063874] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6208.063876] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6208.063878] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6208.063882] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6208.063885] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6208.063888] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6208.063897] Call Trace: [ 6208.063902] [ 6208.063908] ? show_regs.cold.9+0x22/0x2f [ 6208.063919] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6208.063925] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6208.063929] ? watchdog+0x40/0x40 [ 6208.063933] ? hrtimer_interrupt+0x128/0x2c0 [ 6208.063938] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6208.063945] ? apic_timer_interrupt+0xf/0x20 [ 6208.063950] [ 6208.063953] ? native_safe_halt+0x2e/0x50 [ 6208.063957] kvm_wait+0xcd/0xe0 [ 6208.063965] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6208.063974] do_raw_spin_lock+0x177/0x190 [ 6208.063979] _raw_spin_lock+0x21/0x30 [ 6208.063983] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6208.066499] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6208.070781] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6208.070854] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6208.070862] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6208.070864] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6208.070879] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6208.070883] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6208.070889] RAX: 0000000000004b85 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6208.116867] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6208.117388] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6208.117611] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6208.117855] ? woken_wake_function+0x30/0x30 [ 6208.118550] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6208.143478] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6208.143488] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6208.143491] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6208.143493] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6208.143496] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6208.143499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6208.143501] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6208.143510] Call Trace: [ 6208.143516] [ 6208.143521] ? show_regs.cold.9+0x22/0x2f [ 6208.143534] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6208.143540] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6208.143543] ? watchdog+0x40/0x40 [ 6208.143548] ? hrtimer_interrupt+0x128/0x2c0 [ 6208.143552] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6208.143560] ? apic_timer_interrupt+0xf/0x20 [ 6208.143565] [ 6208.143569] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6208.143575] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6208.143580] do_raw_spin_lock+0x177/0x190 [ 6208.143584] _raw_spin_lock+0x21/0x30 [ 6208.143602] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6208.143890] ? do_raw_spin_unlock+0x75/0x190 [ 6208.143897] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6208.145230] ? do_wait_intr+0xf0/0xf0 [ 6208.145239] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6208.145479] mdc_reint+0x2d/0x110 [mdc] [ 6208.145517] mdc_unlink+0x658/0x870 [mdc] [ 6208.145547] lmv_unlink+0x34d/0xbd0 [lmv] [ 6208.145578] ll_unlink+0x24d/0x7f0 [lustre] [ 6208.145877] vfs_unlink+0x202/0x330 [ 6208.145887] do_unlinkat+0x2fa/0x460 [ 6208.145892] __x64_sys_unlink+0x37/0x50 [ 6208.145896] do_syscall_64+0xc1/0x440 [ 6208.145903] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6208.145911] RIP: 0033:0x7f83e5f5c10b [ 6208.145919] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6208.145923] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6208.145937] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6208.145939] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6208.145941] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6208.145943] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6208.145945] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6208.462229] kthread+0x1d1/0x200 [ 6208.465190] ? set_kthread_struct+0x70/0x70 [ 6208.467341] ret_from_fork+0x1f/0x30 [ 6224.056057] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 6224.060622] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6224.069083] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6224.109022] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6224.109035] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6224.109039] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6224.109053] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6224.109058] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6224.109064] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6224.109067] RDX: 0000000000003dd1 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6224.109069] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6224.109072] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6224.109074] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6224.109077] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6224.109080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6224.109082] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6224.109092] Call Trace: [ 6224.109097] [ 6224.109103] ? show_regs.cold.9+0x22/0x2f [ 6224.109114] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6224.109121] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6224.109125] ? watchdog+0x40/0x40 [ 6224.109129] ? hrtimer_interrupt+0x128/0x2c0 [ 6224.109133] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6224.109142] ? apic_timer_interrupt+0xf/0x20 [ 6224.109147] [ 6224.109150] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6224.109156] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6224.109161] do_raw_spin_lock+0x177/0x190 [ 6224.109165] _raw_spin_lock+0x21/0x30 [ 6224.109171] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6224.114229] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6224.114310] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6224.114317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6224.114323] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6224.114776] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6224.114862] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6224.114866] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6224.114871] RAX: 0000000000007d29 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6224.114873] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6224.114876] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6224.114878] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6224.114880] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6224.114884] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6224.114887] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6224.114889] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6224.114897] Call Trace: [ 6224.114903] [ 6224.114908] ? show_regs.cold.9+0x22/0x2f [ 6224.114919] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6224.115563] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6224.115569] ? watchdog+0x40/0x40 [ 6224.115585] ? hrtimer_interrupt+0x128/0x2c0 [ 6224.115590] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6224.115598] ? apic_timer_interrupt+0xf/0x20 [ 6224.115603] [ 6224.115606] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6224.115611] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6224.115617] do_raw_spin_lock+0x177/0x190 [ 6224.115623] _raw_spin_lock+0x21/0x30 [ 6224.115627] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6224.115912] process_one_work+0x2c8/0x700 [ 6224.115920] worker_thread+0x296/0x6e0 [ 6224.115928] ? rescuer_thread+0x570/0x570 [ 6224.115932] kthread+0x1d1/0x200 [ 6224.115937] ? set_kthread_struct+0x70/0x70 [ 6224.115944] ret_from_fork+0x1f/0x30 [ 6224.399147] obd_mod_free+0x46/0x140 [ptlrpc] [ 6224.401601] mdc_commit_open+0x91/0xb0 [mdc] [ 6224.403681] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6224.406905] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6224.410498] after_reply+0xe2c/0x1780 [ptlrpc] [ 6224.412212] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6224.416193] ? __next_timer_interrupt+0x160/0x160 [ 6224.419930] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6224.422781] ? do_wait_intr+0xf0/0xf0 [ 6224.424121] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6224.426441] kthread+0x1d1/0x200 [ 6224.427565] ? set_kthread_struct+0x70/0x70 [ 6224.429672] ret_from_fork+0x1f/0x30 [ 6226.912426] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3713s! [ 6226.922939] Showing busy workqueues and worker pools: [ 6226.926471] workqueue events: flags=0x0 [ 6226.929709] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6226.932206] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6226.935583] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6226.946428] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6226.948718] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6226.955818] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6226.959242] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6226.963994] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6226.971091] workqueue events_long: flags=0x0 [ 6226.972458] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6226.975897] pending: lu_objects_flush [obdclass] [ 6226.979559] workqueue events_unbound: flags=0x2 [ 6226.982686] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6226.986457] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6226.988323] workqueue events_power_efficient: flags=0x80 [ 6226.993363] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6227.000982] pending: neigh_periodic_work, do_cache_clean [ 6227.008017] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6227.012206] pending: neigh_periodic_work, check_lifetime [ 6227.016011] workqueue mm_percpu_wq: flags=0x8 [ 6227.021931] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6227.028640] pending: vmstat_update, lru_add_drain_per_cpu [ 6227.031180] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6227.036293] pending: vmstat_update [ 6227.041017] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6227.049868] pending: lru_add_drain_per_cpu BAR(48) [ 6227.051794] workqueue ipv6_addrconf: flags=0x40008 [ 6227.059464] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6227.063490] pending: addrconf_verify_work [ 6227.065115] workqueue knbd0-recv: flags=0x1a [ 6227.072732] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6227.082785] in-flight: 650:recv_work [ 6227.084095] workqueue ptlrpc_pinger: flags=0x2 [ 6227.085408] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6227.093451] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6227.100094] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3713s workers=4 idle: 36 466 29757 [ 6227.107019] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6217s workers=2 idle: 652 [ 6236.059025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 6236.062436] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6236.062633] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6236.062718] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6236.062724] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6236.062727] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6236.062742] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6236.062746] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6236.062751] RAX: 000000000000372c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6236.062754] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6236.062756] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6236.062758] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6236.062760] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6236.062763] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6236.062830] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6236.062833] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6236.062841] Call Trace: [ 6236.062846] [ 6236.062853] ? show_regs.cold.9+0x22/0x2f [ 6236.062861] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6236.062868] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6236.062873] ? watchdog+0x40/0x40 [ 6236.062877] ? hrtimer_interrupt+0x128/0x2c0 [ 6236.062880] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6236.062888] ? apic_timer_interrupt+0xf/0x20 [ 6236.062892] [ 6236.062895] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6236.062899] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6236.062905] do_raw_spin_lock+0x177/0x190 [ 6236.062916] _raw_spin_lock+0x21/0x30 [ 6236.062922] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6236.063208] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6236.063496] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6236.063779] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6236.063990] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6236.064200] ? woken_wake_function+0x30/0x30 [ 6236.064208] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6236.064562] kthread+0x1d1/0x200 [ 6236.064569] ? set_kthread_struct+0x70/0x70 [ 6236.064572] ret_from_fork+0x1f/0x30 [ 6236.255917] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6236.271079] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6236.277543] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6236.282604] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6236.288969] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6236.302780] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6236.306048] RAX: 000000000000371b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6236.308447] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6236.310958] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6236.314914] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6236.318476] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6236.324265] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6236.328189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6236.332915] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6236.336239] Call Trace: [ 6236.337250] [ 6236.338115] ? show_regs.cold.9+0x22/0x2f [ 6236.339432] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6236.341360] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6236.346074] ? watchdog+0x40/0x40 [ 6236.349303] ? hrtimer_interrupt+0x128/0x2c0 [ 6236.352993] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6236.354433] ? apic_timer_interrupt+0xf/0x20 [ 6236.355747] [ 6236.356266] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6236.364835] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6236.370392] do_raw_spin_lock+0x177/0x190 [ 6236.372432] _raw_spin_lock+0x21/0x30 [ 6236.374450] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6236.377277] ? do_raw_spin_unlock+0x75/0x190 [ 6236.379193] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6236.382321] ? do_wait_intr+0xf0/0xf0 [ 6236.384702] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6236.398975] mdc_reint+0x2d/0x110 [mdc] [ 6236.400425] mdc_unlink+0x658/0x870 [mdc] [ 6236.404008] lmv_unlink+0x34d/0xbd0 [lmv] [ 6236.406365] ll_unlink+0x24d/0x7f0 [lustre] [ 6236.412564] vfs_unlink+0x202/0x330 [ 6236.415620] do_unlinkat+0x2fa/0x460 [ 6236.417408] __x64_sys_unlink+0x37/0x50 [ 6236.421985] do_syscall_64+0xc1/0x440 [ 6236.424339] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6236.429030] RIP: 0033:0x7f83e5f5c10b [ 6236.430811] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6236.440051] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6236.443042] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6236.449422] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6236.453361] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6236.459474] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6236.469720] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6252.056009] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 6252.061190] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6252.062511] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6252.093282] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6252.093310] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6252.093313] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6252.093329] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6252.093333] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6252.093458] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6252.093460] RDX: 0000000000000c56 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6252.093462] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6252.093464] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6252.093466] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6252.093469] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6252.093472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6252.093474] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6252.093483] Call Trace: [ 6252.093488] [ 6252.093494] ? show_regs.cold.9+0x22/0x2f [ 6252.093572] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6252.093580] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6252.093584] ? watchdog+0x40/0x40 [ 6252.093589] ? hrtimer_interrupt+0x128/0x2c0 [ 6252.093593] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6252.093602] ? apic_timer_interrupt+0xf/0x20 [ 6252.093607] [ 6252.093610] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6252.093615] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6252.093620] do_raw_spin_lock+0x177/0x190 [ 6252.093625] _raw_spin_lock+0x21/0x30 [ 6252.093632] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6252.106086] Modules linked in: lustre(O) osp(O) [ 6252.112570] obd_mod_free+0x46/0x140 [ptlrpc] [ 6252.112881] mdc_commit_open+0x91/0xb0 [mdc] [ 6252.112916] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6252.121857] ofd(O) [ 6252.125653] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6252.137284] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6252.137429] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6252.137436] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6252.137442] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6252.137867] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6252.137879] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6252.137883] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6252.137888] RAX: 0000000000001317 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6252.137890] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6252.137893] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6252.137895] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6252.137897] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6252.137901] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6252.137904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6252.137907] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6252.137916] Call Trace: [ 6252.137922] [ 6252.137929] ? show_regs.cold.9+0x22/0x2f [ 6252.138114] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6252.138126] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6252.138131] ? watchdog+0x40/0x40 [ 6252.138139] ? hrtimer_interrupt+0x128/0x2c0 [ 6252.138144] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6252.138153] ? apic_timer_interrupt+0xf/0x20 [ 6252.138158] [ 6252.138161] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6252.138166] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6252.138172] do_raw_spin_lock+0x177/0x190 [ 6252.138176] _raw_spin_lock+0x21/0x30 [ 6252.138181] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6252.138722] process_one_work+0x2c8/0x700 [ 6252.138730] worker_thread+0x296/0x6e0 [ 6252.138734] ? rescuer_thread+0x570/0x570 [ 6252.138738] kthread+0x1d1/0x200 [ 6252.138742] ? set_kthread_struct+0x70/0x70 [ 6252.138746] ret_from_fork+0x1f/0x30 [ 6252.361668] after_reply+0xe2c/0x1780 [ptlrpc] [ 6252.363745] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6252.366643] ? __next_timer_interrupt+0x160/0x160 [ 6252.369291] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6252.371589] ? do_wait_intr+0xf0/0xf0 [ 6252.373324] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6252.376360] kthread+0x1d1/0x200 [ 6252.377693] ? set_kthread_struct+0x70/0x70 [ 6252.379654] ret_from_fork+0x1f/0x30 [ 6257.131578] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3743s! [ 6257.142492] Showing busy workqueues and worker pools: [ 6257.145009] workqueue events: flags=0x0 [ 6257.147286] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6257.150638] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6257.155608] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6257.159649] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6257.165408] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6257.169271] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6257.173415] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6257.177319] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6257.182735] workqueue events_long: flags=0x0 [ 6257.186897] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6257.190931] pending: lu_objects_flush [obdclass] [ 6257.195965] workqueue events_unbound: flags=0x2 [ 6257.198137] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6257.201371] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6257.204012] workqueue events_power_efficient: flags=0x80 [ 6257.206645] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6257.209295] pending: neigh_periodic_work, do_cache_clean [ 6257.212057] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6257.221618] pending: neigh_periodic_work, check_lifetime [ 6257.230969] workqueue mm_percpu_wq: flags=0x8 [ 6257.235957] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6257.244275] pending: vmstat_update, lru_add_drain_per_cpu [ 6257.249383] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6257.254353] pending: vmstat_update [ 6257.257021] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6257.259670] pending: lru_add_drain_per_cpu BAR(48) [ 6257.261778] workqueue ipv6_addrconf: flags=0x40008 [ 6257.262872] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6257.269842] pending: addrconf_verify_work [ 6257.271540] workqueue knbd0-recv: flags=0x1a [ 6257.273196] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6257.281058] in-flight: 650:recv_work [ 6257.282193] workqueue ptlrpc_pinger: flags=0x2 [ 6257.283404] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6257.289420] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6257.299085] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3744s workers=4 idle: 36 466 29757 [ 6257.311758] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6247s workers=2 idle: 652 [ 6264.064008] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 6264.066328] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6264.088656] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6264.088738] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6264.088745] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6264.088747] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6264.088760] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6264.088764] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6264.088768] RAX: 000000000000768a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6264.088771] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6264.088773] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6264.088774] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6264.088776] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6264.088779] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6264.088781] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6264.088784] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6264.088791] Call Trace: [ 6264.088797] [ 6264.088801] ? show_regs.cold.9+0x22/0x2f [ 6264.088809] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6264.088815] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6264.088819] ? watchdog+0x40/0x40 [ 6264.088824] ? hrtimer_interrupt+0x128/0x2c0 [ 6264.088828] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6264.089160] ? apic_timer_interrupt+0xf/0x20 [ 6264.089165] [ 6264.089169] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6264.089173] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6264.089179] do_raw_spin_lock+0x177/0x190 [ 6264.089199] _raw_spin_lock+0x21/0x30 [ 6264.089204] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6264.096414] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6264.096511] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6264.096518] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6264.096521] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6264.096542] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6264.096546] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6264.096550] RAX: 0000000000007295 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6264.096552] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6264.096554] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6264.096556] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6264.096559] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6264.096562] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6264.096569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6264.096571] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6264.096578] Call Trace: [ 6264.096583] [ 6264.096588] ? show_regs.cold.9+0x22/0x2f [ 6264.096602] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6264.096608] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6264.096612] ? watchdog+0x40/0x40 [ 6264.096616] ? hrtimer_interrupt+0x128/0x2c0 [ 6264.096620] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6264.096634] ? apic_timer_interrupt+0xf/0x20 [ 6264.096639] [ 6264.096642] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6264.096647] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6264.096651] do_raw_spin_lock+0x177/0x190 [ 6264.096656] _raw_spin_lock+0x21/0x30 [ 6264.096781] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6264.097307] ? do_raw_spin_unlock+0x75/0x190 [ 6264.097317] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6264.097633] ? do_wait_intr+0xf0/0xf0 [ 6264.097640] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6264.098220] mdc_reint+0x2d/0x110 [mdc] [ 6264.098261] mdc_unlink+0x658/0x870 [mdc] [ 6264.098302] lmv_unlink+0x34d/0xbd0 [lmv] [ 6264.128772] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6264.129087] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6264.129325] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6264.130463] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6264.131343] ? woken_wake_function+0x30/0x30 [ 6264.131353] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6264.131626] kthread+0x1d1/0x200 [ 6264.131632] ? set_kthread_struct+0x70/0x70 [ 6264.131636] ret_from_fork+0x1f/0x30 [ 6264.436032] ll_unlink+0x24d/0x7f0 [lustre] [ 6264.438989] vfs_unlink+0x202/0x330 [ 6264.442965] do_unlinkat+0x2fa/0x460 [ 6264.444870] __x64_sys_unlink+0x37/0x50 [ 6264.446998] do_syscall_64+0xc1/0x440 [ 6264.448829] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6264.451836] RIP: 0033:0x7f83e5f5c10b [ 6264.453914] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6264.465936] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6264.470528] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6264.475223] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6264.479772] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6264.483584] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6264.487970] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6280.060006] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6280.073368] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6280.155562] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6280.176651] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6280.192431] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6280.206798] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6280.220857] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6280.246399] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6280.258785] RAX: 0000000000006f12 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6280.269184] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6280.274651] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6280.277134] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6280.286802] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6280.293165] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6280.302929] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6280.308968] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6280.317426] Call Trace: [ 6280.319750] [ 6280.322063] ? show_regs.cold.9+0x22/0x2f [ 6280.326736] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6280.331253] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6280.333991] ? watchdog+0x40/0x40 [ 6280.338219] ? hrtimer_interrupt+0x128/0x2c0 [ 6280.344910] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6280.346665] ? apic_timer_interrupt+0xf/0x20 [ 6280.356069] [ 6280.356790] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6280.369786] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6280.377388] do_raw_spin_lock+0x177/0x190 [ 6280.384198] _raw_spin_lock+0x21/0x30 [ 6280.385633] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6280.391121] process_one_work+0x2c8/0x700 [ 6280.395940] worker_thread+0x296/0x6e0 [ 6280.403720] ? rescuer_thread+0x570/0x570 [ 6280.408713] kthread+0x1d1/0x200 [ 6280.414211] ? set_kthread_struct+0x70/0x70 [ 6280.419423] ret_from_fork+0x1f/0x30 [ 6284.058013] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 6284.071102] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6284.116468] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6284.126298] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6284.130862] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6284.134680] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6284.144045] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6284.148186] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6284.152110] RDX: 0000000000007b2d RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6284.159289] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6284.163742] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6284.170086] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6284.177491] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6284.186451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6284.193651] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6284.197520] Call Trace: [ 6284.198896] [ 6284.201753] ? show_regs.cold.9+0x22/0x2f [ 6284.204799] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6284.209223] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6284.210817] ? watchdog+0x40/0x40 [ 6284.212913] ? hrtimer_interrupt+0x128/0x2c0 [ 6284.214670] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6284.217121] ? apic_timer_interrupt+0xf/0x20 [ 6284.219037] [ 6284.219961] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6284.222720] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6284.226045] do_raw_spin_lock+0x177/0x190 [ 6284.227893] _raw_spin_lock+0x21/0x30 [ 6284.229661] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6284.231696] obd_mod_free+0x46/0x140 [ptlrpc] [ 6284.234318] mdc_commit_open+0x91/0xb0 [mdc] [ 6284.236422] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6284.244020] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6284.249160] after_reply+0xe2c/0x1780 [ptlrpc] [ 6284.256045] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6284.258816] ? __next_timer_interrupt+0x160/0x160 [ 6284.261154] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6284.265733] ? do_wait_intr+0xf0/0xf0 [ 6284.270029] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6284.276803] kthread+0x1d1/0x200 [ 6284.281015] ? set_kthread_struct+0x70/0x70 [ 6284.283659] ret_from_fork+0x1f/0x30 [ 6287.332262] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3774s! [ 6287.341729] Showing busy workqueues and worker pools: [ 6287.346533] workqueue events: flags=0x0 [ 6287.350322] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6287.361332] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6287.369394] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6287.376531] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6287.383732] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6287.389718] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6287.395322] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6287.401689] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6287.412621] workqueue events_long: flags=0x0 [ 6287.415101] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6287.424020] pending: lu_objects_flush [obdclass] [ 6287.429792] workqueue events_unbound: flags=0x2 [ 6287.433306] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6287.438258] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6287.444643] workqueue events_power_efficient: flags=0x80 [ 6287.449881] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6287.455852] pending: neigh_periodic_work, do_cache_clean [ 6287.462393] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6287.469783] pending: neigh_periodic_work, check_lifetime [ 6287.476151] workqueue mm_percpu_wq: flags=0x8 [ 6287.477768] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6287.489183] pending: vmstat_update, lru_add_drain_per_cpu [ 6287.493056] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6287.499558] pending: vmstat_update [ 6287.504766] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6287.514894] pending: lru_add_drain_per_cpu BAR(48) [ 6287.519445] workqueue ipv6_addrconf: flags=0x40008 [ 6287.524766] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6287.533371] pending: addrconf_verify_work [ 6287.539116] workqueue knbd0-recv: flags=0x1a [ 6287.548235] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6287.560442] in-flight: 650:recv_work [ 6287.562122] workqueue ptlrpc_pinger: flags=0x2 [ 6287.568478] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6287.574308] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6287.580740] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3774s workers=4 idle: 36 466 29757 [ 6287.585935] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6277s workers=2 idle: 652 [ 6292.059024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 6292.061615] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6292.062094] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6292.085197] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6292.085209] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6292.085211] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6292.085223] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6292.085227] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6292.085232] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6292.085234] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6292.085236] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6292.085238] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6292.085241] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6292.085244] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6292.085246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6292.085248] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6292.085256] Call Trace: [ 6292.085261] [ 6292.089628] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix [ 6292.093213] ? show_regs.cold.9+0x22/0x2f [ 6292.093232] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6292.093237] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6292.093241] ? watchdog+0x40/0x40 [ 6292.093245] ? hrtimer_interrupt+0x128/0x2c0 [ 6292.093249] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6292.098413] serio_raw [ 6292.100706] ? apic_timer_interrupt+0xf/0x20 [ 6292.110086] libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6292.110108] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6292.110115] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6292.110118] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6292.110134] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6292.110138] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6292.110142] RAX: 0000000000002d8f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6292.110145] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6292.110147] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6292.110149] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6292.110152] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6292.110154] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6292.110157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6292.110160] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6292.110168] Call Trace: [ 6292.110174] [ 6292.110180] ? show_regs.cold.9+0x22/0x2f [ 6292.113821] [ 6292.117990] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6292.121840] ? native_safe_halt+0x2e/0x50 [ 6292.126634] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6292.126653] ? watchdog+0x40/0x40 [ 6292.126659] ? hrtimer_interrupt+0x128/0x2c0 [ 6292.126664] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6292.126673] ? apic_timer_interrupt+0xf/0x20 [ 6292.126678] [ 6292.126684] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6292.130948] kvm_wait+0xcd/0xe0 [ 6292.134420] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6292.134440] do_raw_spin_lock+0x177/0x190 [ 6292.134446] _raw_spin_lock+0x21/0x30 [ 6292.134455] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6292.134741] ? do_raw_spin_unlock+0x75/0x190 [ 6292.134747] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6292.135603] ? do_wait_intr+0xf0/0xf0 [ 6292.135673] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6292.135877] mdc_reint+0x2d/0x110 [mdc] [ 6292.135922] mdc_unlink+0x658/0x870 [mdc] [ 6292.135951] lmv_unlink+0x34d/0xbd0 [lmv] [ 6292.135983] ll_unlink+0x24d/0x7f0 [lustre] [ 6292.136312] vfs_unlink+0x202/0x330 [ 6292.136321] do_unlinkat+0x2fa/0x460 [ 6292.136326] __x64_sys_unlink+0x37/0x50 [ 6292.136329] do_syscall_64+0xc1/0x440 [ 6292.136336] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6292.136344] RIP: 0033:0x7f83e5f5c10b [ 6292.136351] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6292.136355] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6292.136360] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6292.136363] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6292.136365] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6292.136367] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6292.136370] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6292.402982] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6292.408694] do_raw_spin_lock+0x177/0x190 [ 6292.410532] _raw_spin_lock+0x21/0x30 [ 6292.413416] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6292.416316] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6292.418998] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6292.422129] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6292.425124] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6292.428890] ? woken_wake_function+0x30/0x30 [ 6292.431026] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6292.434418] kthread+0x1d1/0x200 [ 6292.436392] ? set_kthread_struct+0x70/0x70 [ 6292.438438] ret_from_fork+0x1f/0x30 [ 6308.060021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6308.061843] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6308.086141] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6308.091783] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6308.095778] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6308.099767] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6308.102278] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6308.111329] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6308.115023] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6308.118232] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6308.122255] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6308.125808] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6308.129518] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6308.133169] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6308.137047] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6308.139714] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6308.143269] Call Trace: [ 6308.144412] [ 6308.145427] ? show_regs.cold.9+0x22/0x2f [ 6308.147461] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6308.149825] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6308.152214] ? watchdog+0x40/0x40 [ 6308.153806] ? hrtimer_interrupt+0x128/0x2c0 [ 6308.156797] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6308.159076] ? apic_timer_interrupt+0xf/0x20 [ 6308.161148] [ 6308.162091] ? native_safe_halt+0x2e/0x50 [ 6308.163820] kvm_wait+0xcd/0xe0 [ 6308.165254] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6308.167527] do_raw_spin_lock+0x177/0x190 [ 6308.169391] _raw_spin_lock+0x21/0x30 [ 6308.171143] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6308.173803] process_one_work+0x2c8/0x700 [ 6308.175568] worker_thread+0x296/0x6e0 [ 6308.177301] ? rescuer_thread+0x570/0x570 [ 6308.179438] kthread+0x1d1/0x200 [ 6308.180904] ? set_kthread_struct+0x70/0x70 [ 6308.182815] ret_from_fork+0x1f/0x30 [ 6312.056063] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 6312.058253] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6312.072685] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6312.076620] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6312.079502] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6312.080928] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6312.086546] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6312.088539] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6312.091132] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6312.093280] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6312.095444] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6312.097709] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6312.099975] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6312.102705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6312.104427] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6312.106588] Call Trace: [ 6312.107331] [ 6312.108068] ? show_regs.cold.9+0x22/0x2f [ 6312.109422] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6312.110864] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6312.112255] ? watchdog+0x40/0x40 [ 6312.113105] ? hrtimer_interrupt+0x128/0x2c0 [ 6312.114246] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6312.115786] ? apic_timer_interrupt+0xf/0x20 [ 6312.117215] [ 6312.117977] ? native_safe_halt+0x2e/0x50 [ 6312.119217] kvm_wait+0xcd/0xe0 [ 6312.120322] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6312.121982] do_raw_spin_lock+0x177/0x190 [ 6312.123423] _raw_spin_lock+0x21/0x30 [ 6312.124500] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6312.126303] obd_mod_free+0x46/0x140 [ptlrpc] [ 6312.127862] mdc_commit_open+0x91/0xb0 [mdc] [ 6312.129876] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6312.131882] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6312.133636] after_reply+0xe2c/0x1780 [ptlrpc] [ 6312.135365] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6312.137263] ? __next_timer_interrupt+0x160/0x160 [ 6312.139069] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6312.140724] ? do_wait_intr+0xf0/0xf0 [ 6312.142024] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6312.143981] kthread+0x1d1/0x200 [ 6312.144891] ? set_kthread_struct+0x70/0x70 [ 6312.145905] ret_from_fork+0x1f/0x30 [ 6318.049172] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3804s! [ 6318.053554] Showing busy workqueues and worker pools: [ 6318.056165] workqueue events: flags=0x0 [ 6318.058322] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6318.061389] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6318.065689] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6318.069283] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6318.072765] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6318.076536] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6318.081524] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6318.085202] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6318.089338] workqueue events_long: flags=0x0 [ 6318.091631] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6318.095595] pending: lu_objects_flush [obdclass] [ 6318.099299] workqueue events_unbound: flags=0x2 [ 6318.102410] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6318.105963] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6318.109535] workqueue events_power_efficient: flags=0x80 [ 6318.113147] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6318.116657] pending: neigh_periodic_work, do_cache_clean [ 6318.119828] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6318.125717] pending: neigh_periodic_work, check_lifetime [ 6318.128793] workqueue mm_percpu_wq: flags=0x8 [ 6318.131212] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6318.134272] pending: vmstat_update, lru_add_drain_per_cpu [ 6318.136588] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6318.140566] pending: vmstat_update [ 6318.142742] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6318.146875] pending: lru_add_drain_per_cpu BAR(48) [ 6318.149409] workqueue ipv6_addrconf: flags=0x40008 [ 6318.152286] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6318.155532] pending: addrconf_verify_work [ 6318.157592] workqueue knbd0-recv: flags=0x1a [ 6318.159781] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6318.164582] in-flight: 650:recv_work [ 6318.167504] workqueue ptlrpc_pinger: flags=0x2 [ 6318.170246] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6318.174456] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6318.178331] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3804s workers=4 idle: 36 466 29757 [ 6318.183340] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6308s workers=2 idle: 652 [ 6320.061364] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 6320.062018] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 6320.074471] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 [ 6320.077723] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6320.077958] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6320.077965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6320.077967] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6320.077979] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6320.077982] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6320.077985] RAX: 0000000000005386 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6320.077987] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6320.077988] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6320.077990] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6320.077991] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6320.077994] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6320.077996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6320.077997] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6320.078004] Call Trace: [ 6320.078009] [ 6320.078015] ? show_regs.cold.9+0x22/0x2f [ 6320.078023] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6320.078028] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6320.078031] ? watchdog+0x40/0x40 [ 6320.078034] ? hrtimer_interrupt+0x128/0x2c0 [ 6320.078036] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6320.078042] ? apic_timer_interrupt+0xf/0x20 [ 6320.078046] [ 6320.078048] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6320.078050] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6320.078053] do_raw_spin_lock+0x177/0x190 [ 6320.078056] _raw_spin_lock+0x21/0x30 [ 6320.078061] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6320.078326] ? do_raw_spin_unlock+0x75/0x190 [ 6320.078332] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6320.078541] ? do_wait_intr+0xf0/0xf0 [ 6320.078547] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6320.080039] mdc_reint+0x2d/0x110 [mdc] [ 6320.080077] mdc_unlink+0x658/0x870 [mdc] [ 6320.080104] lmv_unlink+0x34d/0xbd0 [lmv] [ 6320.080141] ll_unlink+0x24d/0x7f0 [lustre] [ 6320.080239] vfs_unlink+0x202/0x330 [ 6320.080248] do_unlinkat+0x2fa/0x460 [ 6320.080253] __x64_sys_unlink+0x37/0x50 [ 6320.080257] do_syscall_64+0xc1/0x440 [ 6320.080265] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6320.080272] RIP: 0033:0x7f83e5f5c10b [ 6320.080280] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6320.080284] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6320.080289] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6320.080291] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6320.080293] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6320.080295] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6320.080297] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6320.336358] auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6320.355873] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6320.373518] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6320.381724] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6320.383414] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6320.409307] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6320.419244] RAX: 0000000000007f85 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6320.431424] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6320.433519] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6320.449776] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6320.458348] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6320.464993] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6320.470700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6320.488741] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6320.499752] Call Trace: [ 6320.500811] [ 6320.501771] ? show_regs.cold.9+0x22/0x2f [ 6320.506530] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6320.508747] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6320.517709] ? watchdog+0x40/0x40 [ 6320.519027] ? hrtimer_interrupt+0x128/0x2c0 [ 6320.522552] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6320.526978] ? apic_timer_interrupt+0xf/0x20 [ 6320.529906] [ 6320.530936] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6320.537826] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6320.546381] do_raw_spin_lock+0x177/0x190 [ 6320.551281] _raw_spin_lock+0x21/0x30 [ 6320.554534] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6320.557418] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6320.559497] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6320.564127] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6320.575532] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6320.577627] ? woken_wake_function+0x30/0x30 [ 6320.581562] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6320.593705] kthread+0x1d1/0x200 [ 6320.596661] ? set_kthread_struct+0x70/0x70 [ 6320.603387] ret_from_fork+0x1f/0x30 [ 6336.060035] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6336.063840] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6336.084169] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6336.087665] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6336.090412] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6336.092295] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6336.094172] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6336.102925] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6336.111469] RAX: 0000000000003281 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6336.114309] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6336.117917] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6336.123199] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6336.126063] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6336.129120] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6336.132969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6336.135549] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6336.138329] Call Trace: [ 6336.139328] [ 6336.140182] ? show_regs.cold.9+0x22/0x2f [ 6336.145262] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6336.149112] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6336.151567] ? watchdog+0x40/0x40 [ 6336.154149] ? hrtimer_interrupt+0x128/0x2c0 [ 6336.157089] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6336.159547] ? apic_timer_interrupt+0xf/0x20 [ 6336.166620] [ 6336.167344] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6336.170152] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6336.172351] do_raw_spin_lock+0x177/0x190 [ 6336.174283] _raw_spin_lock+0x21/0x30 [ 6336.176555] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6336.180411] process_one_work+0x2c8/0x700 [ 6336.185716] worker_thread+0x296/0x6e0 [ 6336.186944] ? rescuer_thread+0x570/0x570 [ 6336.190410] kthread+0x1d1/0x200 [ 6336.192895] ? set_kthread_struct+0x70/0x70 [ 6336.195822] ret_from_fork+0x1f/0x30 [ 6344.056025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 6344.062118] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6344.100814] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6344.109831] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6344.115853] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6344.120728] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6344.132505] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6344.139596] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6344.153576] RDX: 0000000000006f8f RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6344.158486] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6344.161419] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6344.165284] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6344.167923] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6344.170974] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6344.172943] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6344.177849] Call Trace: [ 6344.178972] [ 6344.179841] ? show_regs.cold.9+0x22/0x2f [ 6344.183870] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6344.187082] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6344.191597] ? watchdog+0x40/0x40 [ 6344.193544] ? hrtimer_interrupt+0x128/0x2c0 [ 6344.195348] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6344.197909] ? apic_timer_interrupt+0xf/0x20 [ 6344.199823] [ 6344.200738] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6344.203090] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6344.205547] do_raw_spin_lock+0x177/0x190 [ 6344.207303] _raw_spin_lock+0x21/0x30 [ 6344.208854] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6344.211965] obd_mod_free+0x46/0x140 [ptlrpc] [ 6344.215271] mdc_commit_open+0x91/0xb0 [mdc] [ 6344.217097] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6344.220921] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6344.223503] after_reply+0xe2c/0x1780 [ptlrpc] [ 6344.225753] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6344.228025] ? __next_timer_interrupt+0x160/0x160 [ 6344.229810] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6344.231091] ? do_wait_intr+0xf0/0xf0 [ 6344.232011] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6344.235689] kthread+0x1d1/0x200 [ 6344.238333] ? set_kthread_struct+0x70/0x70 [ 6344.241096] ret_from_fork+0x1f/0x30 [ 6348.059279] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 6348.062294] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 6348.065079] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6348.065157] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6348.065165] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6348.065168] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6348.065181] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6348.065185] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6348.065190] RAX: 0000000000007a87 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6348.065192] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6348.065194] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6348.065196] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6348.065199] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6348.065202] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6348.065205] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6348.065207] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6348.065216] Call Trace: [ 6348.065221] [ 6348.065227] ? show_regs.cold.9+0x22/0x2f [ 6348.065237] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6348.065243] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6348.065247] ? watchdog+0x40/0x40 [ 6348.065252] ? hrtimer_interrupt+0x128/0x2c0 [ 6348.065256] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6348.065510] ? apic_timer_interrupt+0xf/0x20 [ 6348.065514] [ 6348.065517] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6348.065522] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6348.065528] do_raw_spin_lock+0x177/0x190 [ 6348.065534] _raw_spin_lock+0x21/0x30 [ 6348.065539] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6348.065810] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6348.066003] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6348.066222] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6348.066692] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6348.066899] ? woken_wake_function+0x30/0x30 [ 6348.066909] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6348.067122] kthread+0x1d1/0x200 [ 6348.067129] ? set_kthread_struct+0x70/0x70 [ 6348.067132] ret_from_fork+0x1f/0x30 [ 6348.235685] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6348.250688] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6348.254023] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6348.256496] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6348.258524] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6348.263589] BUG: workqueue lockup - pool [ 6348.264224] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6348.264302] RAX: 0000000000005448 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6348.264305] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6348.264307] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6348.264309] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6348.264311] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6348.264313] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6348.264316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6348.264318] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6348.264326] Call Trace: [ 6348.264333] [ 6348.264338] ? show_regs.cold.9+0x22/0x2f [ 6348.264350] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6348.264356] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6348.264359] ? watchdog+0x40/0x40 [ 6348.264363] ? hrtimer_interrupt+0x128/0x2c0 [ 6348.264379] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6348.264388] ? apic_timer_interrupt+0xf/0x20 [ 6348.264392] [ 6348.264396] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6348.264402] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6348.264406] do_raw_spin_lock+0x177/0x190 [ 6348.264411] _raw_spin_lock+0x21/0x30 [ 6348.264418] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6348.264832] ? do_raw_spin_unlock+0x75/0x190 [ 6348.264839] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6348.265049] ? do_wait_intr+0xf0/0xf0 [ 6348.265057] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6348.265424] mdc_reint+0x2d/0x110 [mdc] [ 6348.265463] mdc_unlink+0x658/0x870 [mdc] [ 6348.265493] lmv_unlink+0x34d/0xbd0 [lmv] [ 6348.265520] ll_unlink+0x24d/0x7f0 [lustre] [ 6348.265773] vfs_unlink+0x202/0x330 [ 6348.265782] do_unlinkat+0x2fa/0x460 [ 6348.265787] __x64_sys_unlink+0x37/0x50 [ 6348.265790] do_syscall_64+0xc1/0x440 [ 6348.265796] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6348.265803] RIP: 0033:0x7f83e5f5c10b [ 6348.265811] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6348.265815] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6348.265820] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6348.265823] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6348.265825] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6348.265827] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6348.265830] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6348.476154] cpus=0-3 flags=0x4 nice=0 stuck for 3835s! [ 6348.481474] Showing busy workqueues and worker pools: [ 6348.487047] workqueue events: flags=0x0 [ 6348.489013] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6348.492966] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6348.498301] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6348.503197] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6348.508009] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6348.511771] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6348.515665] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6348.518988] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6348.523376] workqueue events_long: flags=0x0 [ 6348.525313] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6348.528471] pending: lu_objects_flush [obdclass] [ 6348.531208] workqueue events_unbound: flags=0x2 [ 6348.533132] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6348.535924] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6348.538554] workqueue events_power_efficient: flags=0x80 [ 6348.541301] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6348.544380] pending: neigh_periodic_work, do_cache_clean [ 6348.546404] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6348.548615] pending: neigh_periodic_work, check_lifetime [ 6348.551210] workqueue mm_percpu_wq: flags=0x8 [ 6348.552938] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6348.558858] pending: vmstat_update, lru_add_drain_per_cpu [ 6348.560451] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6348.561929] pending: vmstat_update [ 6348.566396] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6348.578252] pending: lru_add_drain_per_cpu BAR(48) [ 6348.582469] workqueue ipv6_addrconf: flags=0x40008 [ 6348.585576] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6348.589473] pending: addrconf_verify_work [ 6348.591598] workqueue knbd0-recv: flags=0x1a [ 6348.596126] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6348.608687] in-flight: 650:recv_work [ 6348.610813] workqueue ptlrpc_pinger: flags=0x2 [ 6348.614340] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6348.617350] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6348.620506] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3835s workers=4 idle: 36 466 29757 [ 6348.626058] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6338s workers=2 idle: 652 [ 6361.060024] rcu: INFO: rcu_sched self-detected stall on CPU [ 6361.066421] rcu: 0-....: (2779446 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 6361.077212] (t=4380118 jiffies g=262441 q=58465) [ 6361.084386] rcu: rcu_sched kthread starved for 3849312 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 6361.097287] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6361.102818] rcu: RCU grace-period kthread stack dump: [ 6361.105426] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6361.110576] Call Trace: [ 6361.111770] __schedule+0x351/0xcb0 [ 6361.113357] schedule+0xc0/0x180 [ 6361.114718] schedule_timeout+0xb4/0x190 [ 6361.116627] ? do_raw_spin_unlock+0x75/0x190 [ 6361.118690] ? __next_timer_interrupt+0x160/0x160 [ 6361.120819] ? prepare_to_swait_event+0xf3/0x140 [ 6361.123573] rcu_gp_kthread+0x8dd/0x15b0 [ 6361.125661] ? force_qs_rnp+0x2c0/0x2c0 [ 6361.127788] kthread+0x1d1/0x200 [ 6361.129402] ? set_kthread_struct+0x70/0x70 [ 6361.131811] ret_from_fork+0x1f/0x30 [ 6361.134885] rcu: Stack dump where RCU GP kthread last ran: [ 6361.137027] Sending NMI from CPU 0 to CPUs 3: [ 6361.139927] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [ 6361.144844] NMI backtrace for cpu 0 [ 6361.148702] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6361.156536] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6361.160631] Call Trace: [ 6361.162748] [ 6361.163625] ? dump_stack+0xbb/0x10e [ 6361.165802] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6361.168517] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6361.170989] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6361.173534] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6361.176410] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6361.179395] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6361.181287] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6361.184007] ? update_process_times+0x6d/0xc0 [ 6361.186630] ? tick_sched_handle.isra.18+0x27/0x90 [ 6361.188943] ? tick_sched_timer+0x53/0xb0 [ 6361.190958] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6361.193309] ? tick_sched_handle.isra.18+0x90/0x90 [ 6361.196399] ? hrtimer_interrupt+0x128/0x2c0 [ 6361.199214] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6361.202760] ? apic_timer_interrupt+0xf/0x20 [ 6361.205357] [ 6361.206782] ? native_safe_halt+0x2e/0x50 [ 6361.209314] ? kvm_wait+0xcd/0xe0 [ 6361.211624] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6361.214391] ? do_raw_spin_lock+0x177/0x190 [ 6361.216802] ? _raw_spin_lock+0x21/0x30 [ 6361.218938] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6361.221180] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6361.223348] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6361.225155] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6361.228942] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6361.232182] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6361.234601] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6361.237288] ? __next_timer_interrupt+0x160/0x160 [ 6361.239453] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6361.242185] ? do_wait_intr+0xf0/0xf0 [ 6361.243931] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6361.246752] ? kthread+0x1d1/0x200 [ 6361.249434] ? set_kthread_struct+0x70/0x70 [ 6361.251205] ? ret_from_fork+0x1f/0x30 [ 6364.060164] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6364.063027] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6364.089389] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6364.094599] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6364.097798] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6364.100109] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6364.102530] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6364.110554] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6364.113581] RAX: 0000000000006d0c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6364.116593] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6364.119310] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6364.122239] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6364.124967] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6364.128516] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6364.133408] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6364.136385] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6364.139657] Call Trace: [ 6364.140648] [ 6364.141914] ? show_regs.cold.9+0x22/0x2f [ 6364.143755] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6364.146193] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6364.148327] ? watchdog+0x40/0x40 [ 6364.150793] ? hrtimer_interrupt+0x128/0x2c0 [ 6364.152553] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6364.154434] ? apic_timer_interrupt+0xf/0x20 [ 6364.155905] [ 6364.156809] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6364.158892] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6364.160846] do_raw_spin_lock+0x177/0x190 [ 6364.162176] _raw_spin_lock+0x21/0x30 [ 6364.164090] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6364.167738] process_one_work+0x2c8/0x700 [ 6364.169221] worker_thread+0x296/0x6e0 [ 6364.170921] ? rescuer_thread+0x570/0x570 [ 6364.172783] kthread+0x1d1/0x200 [ 6364.174372] ? set_kthread_struct+0x70/0x70 [ 6364.176334] ret_from_fork+0x1f/0x30 [ 6376.059031] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 6376.061963] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6376.088231] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6376.096537] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6376.103532] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6376.108267] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6376.119369] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6376.121742] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6376.128032] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6376.132852] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6376.138356] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6376.141612] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6376.146651] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6376.150762] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6376.153804] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6376.157934] Call Trace: [ 6376.159603] [ 6376.160870] ? show_regs.cold.9+0x22/0x2f [ 6376.164313] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6376.165920] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6376.174423] ? watchdog+0x40/0x40 [ 6376.175547] ? hrtimer_interrupt+0x128/0x2c0 [ 6376.177850] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6376.180086] ? apic_timer_interrupt+0xf/0x20 [ 6376.182310] [ 6376.183305] ? native_safe_halt+0x2e/0x50 [ 6376.185293] kvm_wait+0xcd/0xe0 [ 6376.186655] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6376.189706] do_raw_spin_lock+0x177/0x190 [ 6376.191720] _raw_spin_lock+0x21/0x30 [ 6376.193667] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6376.197229] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6376.200214] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6376.202709] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6376.205493] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6376.208168] ? woken_wake_function+0x30/0x30 [ 6376.210081] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6376.213255] kthread+0x1d1/0x200 [ 6376.214894] ? set_kthread_struct+0x70/0x70 [ 6376.216559] ret_from_fork+0x1f/0x30 [ 6378.976553] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3865s! [ 6378.987049] Showing busy workqueues and worker pools: [ 6378.989427] workqueue events: flags=0x0 [ 6378.992023] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6378.995731] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6379.001153] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6379.006057] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6379.009703] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6379.013056] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6379.019221] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6379.023621] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6379.028600] workqueue events_long: flags=0x0 [ 6379.030697] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6379.035254] pending: lu_objects_flush [obdclass] [ 6379.038596] workqueue events_unbound: flags=0x2 [ 6379.041342] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6379.044973] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6379.048240] workqueue events_power_efficient: flags=0x80 [ 6379.051035] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6379.055206] pending: neigh_periodic_work, do_cache_clean [ 6379.058736] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6379.062549] pending: neigh_periodic_work, check_lifetime [ 6379.066643] workqueue mm_percpu_wq: flags=0x8 [ 6379.070124] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6379.074022] pending: vmstat_update, lru_add_drain_per_cpu [ 6379.077349] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6379.080486] pending: vmstat_update [ 6379.082586] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6379.086176] pending: lru_add_drain_per_cpu BAR(48) [ 6379.088896] workqueue ipv6_addrconf: flags=0x40008 [ 6379.091532] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6379.095022] pending: addrconf_verify_work [ 6379.097404] workqueue knbd0-recv: flags=0x1a [ 6379.099722] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6379.103671] in-flight: 650:recv_work [ 6379.105856] workqueue ptlrpc_pinger: flags=0x2 [ 6379.108253] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6379.111611] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6379.114640] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3865s workers=4 idle: 36 466 29757 [ 6379.118386] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6369s workers=2 idle: 652 [ 6388.062094] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 6388.066645] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6388.092120] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6388.097685] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6388.102648] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6388.106536] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6388.116143] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6388.120697] RAX: 000000000000667d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6388.122510] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6388.125592] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6388.130641] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6388.140100] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6388.142672] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6388.146472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6388.148424] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6388.151553] Call Trace: [ 6388.152761] [ 6388.157280] ? show_regs.cold.9+0x22/0x2f [ 6388.158770] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6388.161691] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6388.165442] ? watchdog+0x40/0x40 [ 6388.167495] ? hrtimer_interrupt+0x128/0x2c0 [ 6388.169891] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6388.171326] ? apic_timer_interrupt+0xf/0x20 [ 6388.172657] [ 6388.173632] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6388.177411] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6388.180765] do_raw_spin_lock+0x177/0x190 [ 6388.184559] _raw_spin_lock+0x21/0x30 [ 6388.188033] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6388.194522] ? do_raw_spin_unlock+0x75/0x190 [ 6388.199688] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6388.203768] ? do_wait_intr+0xf0/0xf0 [ 6388.204958] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6388.211554] mdc_reint+0x2d/0x110 [mdc] [ 6388.213407] mdc_unlink+0x658/0x870 [mdc] [ 6388.215149] lmv_unlink+0x34d/0xbd0 [lmv] [ 6388.219425] ll_unlink+0x24d/0x7f0 [lustre] [ 6388.222699] vfs_unlink+0x202/0x330 [ 6388.225370] do_unlinkat+0x2fa/0x460 [ 6388.226660] __x64_sys_unlink+0x37/0x50 [ 6388.228492] do_syscall_64+0xc1/0x440 [ 6388.230276] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6388.231608] RIP: 0033:0x7f83e5f5c10b [ 6388.232642] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6388.246581] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6388.253421] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6388.257169] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6388.262343] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6388.264581] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6388.267344] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6392.060018] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 6392.065829] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6392.098471] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6392.111415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6392.121702] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6392.127619] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6392.131961] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6392.145531] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6392.162538] RAX: 0000000000007f8a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6392.167705] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6392.173091] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6392.179585] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6392.183737] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6392.188613] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6392.197431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6392.202981] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6392.208349] Call Trace: [ 6392.209566] [ 6392.211485] ? show_regs.cold.9+0x22/0x2f [ 6392.213559] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6392.218593] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6392.226268] ? watchdog+0x40/0x40 [ 6392.233716] ? hrtimer_interrupt+0x128/0x2c0 [ 6392.244587] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6392.247518] ? apic_timer_interrupt+0xf/0x20 [ 6392.254865] [ 6392.257402] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6392.267479] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6392.275789] do_raw_spin_lock+0x177/0x190 [ 6392.277347] _raw_spin_lock+0x21/0x30 [ 6392.286480] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6392.291074] process_one_work+0x2c8/0x700 [ 6392.296428] worker_thread+0x296/0x6e0 [ 6392.304862] ? rescuer_thread+0x570/0x570 [ 6392.307757] kthread+0x1d1/0x200 [ 6392.309575] ? set_kthread_struct+0x70/0x70 [ 6392.311821] ret_from_fork+0x1f/0x30 [ 6404.058525] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 6404.060122] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 6404.062193] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6404.062278] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6404.062288] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6404.062291] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6404.062306] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6404.062313] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6404.062317] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6404.062320] RDX: 0000000000006735 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6404.062323] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6404.062325] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6404.062330] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6404.062333] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6404.062336] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6404.062338] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6404.062348] Call Trace: [ 6404.062352] [ 6404.062357] ? show_regs.cold.9+0x22/0x2f [ 6404.062366] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6404.062372] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6404.062377] ? watchdog+0x40/0x40 [ 6404.062384] ? hrtimer_interrupt+0x128/0x2c0 [ 6404.062444] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6404.062453] ? apic_timer_interrupt+0xf/0x20 [ 6404.062458] [ 6404.062461] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6404.062466] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6404.062470] do_raw_spin_lock+0x177/0x190 [ 6404.062475] _raw_spin_lock+0x21/0x30 [ 6404.062480] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6404.062773] obd_mod_free+0x46/0x140 [ptlrpc] [ 6404.063116] mdc_commit_open+0x91/0xb0 [mdc] [ 6404.063149] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6404.063333] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6404.063615] after_reply+0xe2c/0x1780 [ptlrpc] [ 6404.063864] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6404.068622] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6404.068704] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6404.068710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6404.068712] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6404.068725] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6404.068729] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6404.068733] RAX: 0000000000007f30 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6404.068736] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6404.068738] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6404.068740] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6404.068742] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6404.068745] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6404.068748] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6404.068750] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6404.068758] Call Trace: [ 6404.068763] [ 6404.068769] ? show_regs.cold.9+0x22/0x2f [ 6404.068779] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6404.068787] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6404.068792] ? watchdog+0x40/0x40 [ 6404.068796] ? hrtimer_interrupt+0x128/0x2c0 [ 6404.068800] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6404.068808] ? apic_timer_interrupt+0xf/0x20 [ 6404.068813] [ 6404.068816] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6404.068820] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6404.068834] do_raw_spin_lock+0x177/0x190 [ 6404.068840] _raw_spin_lock+0x21/0x30 [ 6404.068845] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6404.069315] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6404.069677] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6404.070332] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6404.070735] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6404.098967] ? __next_timer_interrupt+0x160/0x160 [ 6404.098988] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6404.099943] ? do_wait_intr+0xf0/0xf0 [ 6404.099955] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6404.108176] ? woken_wake_function+0x30/0x30 [ 6404.108198] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6404.108712] kthread+0x1d1/0x200 [ 6404.108719] ? set_kthread_struct+0x70/0x70 [ 6404.108723] ret_from_fork+0x1f/0x30 [ 6404.345860] kthread+0x1d1/0x200 [ 6404.347487] ? set_kthread_struct+0x70/0x70 [ 6404.349650] ret_from_fork+0x1f/0x30 [ 6409.184091] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3895s! [ 6409.187374] Showing busy workqueues and worker pools: [ 6409.189679] workqueue events: flags=0x0 [ 6409.191335] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6409.195518] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6409.198905] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6409.201525] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6409.204100] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6409.206464] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6409.209678] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6409.211926] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6409.215272] workqueue events_long: flags=0x0 [ 6409.217136] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6409.220255] pending: lu_objects_flush [obdclass] [ 6409.222732] workqueue events_unbound: flags=0x2 [ 6409.224463] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6409.227338] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6409.229876] workqueue events_power_efficient: flags=0x80 [ 6409.232443] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6409.235469] pending: neigh_periodic_work, do_cache_clean [ 6409.238167] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6409.241071] pending: neigh_periodic_work, check_lifetime [ 6409.245860] workqueue mm_percpu_wq: flags=0x8 [ 6409.248381] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6409.251462] pending: vmstat_update, lru_add_drain_per_cpu [ 6409.253806] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6409.257107] pending: vmstat_update [ 6409.258588] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6409.261975] pending: lru_add_drain_per_cpu BAR(48) [ 6409.264493] workqueue ipv6_addrconf: flags=0x40008 [ 6409.267349] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6409.271351] pending: addrconf_verify_work [ 6409.273842] workqueue knbd0-recv: flags=0x1a [ 6409.276412] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6409.289987] in-flight: 650:recv_work [ 6409.291438] workqueue ptlrpc_pinger: flags=0x2 [ 6409.300168] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6409.310520] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6409.312373] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3896s workers=4 idle: 36 466 29757 [ 6409.328440] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6399s workers=2 idle: 652 [ 6416.062028] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6416.067294] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6416.096357] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6416.102544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6416.109510] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6416.112225] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6416.121496] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6416.125154] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6416.129784] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6416.133627] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6416.137299] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6416.142573] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6416.147220] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6416.151406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6416.153951] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6416.156315] Call Trace: [ 6416.157392] [ 6416.158386] ? show_regs.cold.9+0x22/0x2f [ 6416.159999] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6416.161617] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6416.163436] ? watchdog+0x40/0x40 [ 6416.164874] ? hrtimer_interrupt+0x128/0x2c0 [ 6416.167071] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6416.168889] ? apic_timer_interrupt+0xf/0x20 [ 6416.170612] [ 6416.171464] ? native_safe_halt+0x2e/0x50 [ 6416.173146] kvm_wait+0xcd/0xe0 [ 6416.174329] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6416.176493] do_raw_spin_lock+0x177/0x190 [ 6416.177870] _raw_spin_lock+0x21/0x30 [ 6416.178964] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6416.181080] ? do_raw_spin_unlock+0x75/0x190 [ 6416.183110] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6416.185078] ? do_wait_intr+0xf0/0xf0 [ 6416.186323] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6416.188397] mdc_reint+0x2d/0x110 [mdc] [ 6416.190185] mdc_unlink+0x658/0x870 [mdc] [ 6416.191779] lmv_unlink+0x34d/0xbd0 [lmv] [ 6416.193394] ll_unlink+0x24d/0x7f0 [lustre] [ 6416.196201] vfs_unlink+0x202/0x330 [ 6416.197198] do_unlinkat+0x2fa/0x460 [ 6416.198431] __x64_sys_unlink+0x37/0x50 [ 6416.200733] do_syscall_64+0xc1/0x440 [ 6416.202309] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6416.204230] RIP: 0033:0x7f83e5f5c10b [ 6416.205717] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6416.212904] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6416.216739] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6416.220614] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6416.225623] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6416.229271] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6416.233060] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6420.060027] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 6420.066325] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6420.136549] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6420.143457] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6420.152042] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6420.155725] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6420.158089] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6420.175590] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6420.182245] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6420.186543] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6420.191643] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6420.201779] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6420.216579] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6420.228292] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6420.237684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6420.245416] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6420.252460] Call Trace: [ 6420.263489] [ 6420.266711] ? show_regs.cold.9+0x22/0x2f [ 6420.279804] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6420.282769] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6420.284466] ? watchdog+0x40/0x40 [ 6420.292334] ? hrtimer_interrupt+0x128/0x2c0 [ 6420.294606] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6420.296653] ? apic_timer_interrupt+0xf/0x20 [ 6420.301467] [ 6420.302692] ? native_safe_halt+0x2e/0x50 [ 6420.304530] kvm_wait+0xcd/0xe0 [ 6420.306564] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6420.309442] do_raw_spin_lock+0x177/0x190 [ 6420.311707] _raw_spin_lock+0x21/0x30 [ 6420.313600] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6420.316233] process_one_work+0x2c8/0x700 [ 6420.318274] worker_thread+0x296/0x6e0 [ 6420.320096] ? rescuer_thread+0x570/0x570 [ 6420.321835] kthread+0x1d1/0x200 [ 6420.324551] ? set_kthread_struct+0x70/0x70 [ 6420.326574] ret_from_fork+0x1f/0x30 [ 6432.065879] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 6432.074492] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6432.156672] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6432.178349] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6432.193615] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6432.197014] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6432.227578] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6432.236632] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6432.246645] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6432.250898] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6432.257717] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6432.269993] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6432.280680] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6432.296510] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6432.306713] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6432.317929] Call Trace: [ 6432.320047] [ 6432.324581] ? show_regs.cold.9+0x22/0x2f [ 6432.330684] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6432.336657] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6432.342121] ? watchdog+0x40/0x40 [ 6432.349524] ? hrtimer_interrupt+0x128/0x2c0 [ 6432.354346] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6432.360915] ? apic_timer_interrupt+0xf/0x20 [ 6432.365498] [ 6432.368573] ? native_safe_halt+0x2e/0x50 [ 6432.373134] kvm_wait+0xcd/0xe0 [ 6432.376748] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6432.385594] do_raw_spin_lock+0x177/0x190 [ 6432.391147] _raw_spin_lock+0x21/0x30 [ 6432.395644] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6432.402223] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6432.406711] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6432.411190] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6432.434776] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6432.449977] ? woken_wake_function+0x30/0x30 [ 6432.459996] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6432.473106] kthread+0x1d1/0x200 [ 6432.474660] ? set_kthread_struct+0x70/0x70 [ 6432.481329] ret_from_fork+0x1f/0x30 [ 6436.056965] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 6436.064780] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6436.088586] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6436.094025] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6436.098272] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6436.101086] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6436.110100] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6436.114363] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6436.117878] RDX: 0000000000007b06 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6436.123920] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6436.137467] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6436.145594] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6436.152576] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6436.159955] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6436.166148] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6436.172954] Call Trace: [ 6436.176042] [ 6436.177961] ? show_regs.cold.9+0x22/0x2f [ 6436.182928] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6436.186273] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6436.189329] ? watchdog+0x40/0x40 [ 6436.191900] ? hrtimer_interrupt+0x128/0x2c0 [ 6436.200904] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6436.204111] ? apic_timer_interrupt+0xf/0x20 [ 6436.210478] [ 6436.211211] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6436.214600] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6436.217362] do_raw_spin_lock+0x177/0x190 [ 6436.219577] _raw_spin_lock+0x21/0x30 [ 6436.222696] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6436.225942] obd_mod_free+0x46/0x140 [ptlrpc] [ 6436.228534] mdc_commit_open+0x91/0xb0 [mdc] [ 6436.230854] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6436.233816] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6436.236583] after_reply+0xe2c/0x1780 [ptlrpc] [ 6436.238297] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6436.240110] ? __next_timer_interrupt+0x160/0x160 [ 6436.242110] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6436.243890] ? do_wait_intr+0xf0/0xf0 [ 6436.245332] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6436.249146] kthread+0x1d1/0x200 [ 6436.250089] ? set_kthread_struct+0x70/0x70 [ 6436.257234] ret_from_fork+0x1f/0x30 [ 6439.393906] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3926s! [ 6439.407572] Showing busy workqueues and worker pools: [ 6439.409544] workqueue events: flags=0x0 [ 6439.415062] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6439.418885] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6439.435037] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6439.442806] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6439.449261] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6439.456238] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6439.468021] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6439.475254] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6439.490535] workqueue events_long: flags=0x0 [ 6439.491874] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6439.498216] pending: lu_objects_flush [obdclass] [ 6439.503194] workqueue events_unbound: flags=0x2 [ 6439.508026] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6439.511144] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6439.514760] workqueue events_power_efficient: flags=0x80 [ 6439.527661] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6439.533037] pending: neigh_periodic_work, do_cache_clean [ 6439.538138] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6439.541884] pending: neigh_periodic_work, check_lifetime [ 6439.545988] workqueue mm_percpu_wq: flags=0x8 [ 6439.553030] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6439.560407] pending: vmstat_update, lru_add_drain_per_cpu [ 6439.569058] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6439.572790] pending: vmstat_update [ 6439.576705] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6439.585296] pending: lru_add_drain_per_cpu BAR(48) [ 6439.589030] workqueue ipv6_addrconf: flags=0x40008 [ 6439.595125] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6439.598120] pending: addrconf_verify_work [ 6439.605571] workqueue knbd0-recv: flags=0x1a [ 6439.609834] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6439.613818] in-flight: 650:recv_work [ 6439.615956] workqueue ptlrpc_pinger: flags=0x2 [ 6439.618679] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6439.629563] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6439.634602] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3926s workers=4 idle: 36 466 29757 [ 6439.641628] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6429s workers=2 idle: 652 [ 6444.062569] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6444.067579] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6444.096113] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6444.102995] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6444.110132] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6444.117365] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6444.145518] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6444.159747] RAX: 0000000000006117 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6444.168796] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6444.178921] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6444.187777] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6444.198073] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6444.206604] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6444.221736] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6444.229611] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6444.240702] Call Trace: [ 6444.242606] [ 6444.245718] ? show_regs.cold.9+0x22/0x2f [ 6444.250005] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6444.255646] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6444.258251] ? watchdog+0x40/0x40 [ 6444.267757] ? hrtimer_interrupt+0x128/0x2c0 [ 6444.273593] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6444.276992] ? apic_timer_interrupt+0xf/0x20 [ 6444.282862] [ 6444.287005] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6444.297530] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6444.304001] do_raw_spin_lock+0x177/0x190 [ 6444.305549] _raw_spin_lock+0x21/0x30 [ 6444.306828] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6444.321394] ? do_raw_spin_unlock+0x75/0x190 [ 6444.325472] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6444.331311] ? do_wait_intr+0xf0/0xf0 [ 6444.332776] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6444.336910] mdc_reint+0x2d/0x110 [mdc] [ 6444.338528] mdc_unlink+0x658/0x870 [mdc] [ 6444.340381] lmv_unlink+0x34d/0xbd0 [lmv] [ 6444.344342] ll_unlink+0x24d/0x7f0 [lustre] [ 6444.348833] vfs_unlink+0x202/0x330 [ 6444.350788] do_unlinkat+0x2fa/0x460 [ 6444.352130] __x64_sys_unlink+0x37/0x50 [ 6444.353189] do_syscall_64+0xc1/0x440 [ 6444.354247] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6444.356022] RIP: 0033:0x7f83e5f5c10b [ 6444.357423] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6444.369480] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6444.374520] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6444.377980] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6444.381248] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6444.384319] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6444.387335] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6448.060306] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 6448.070066] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6448.115016] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6448.127550] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6448.135318] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6448.139039] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6448.148051] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6448.164163] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6448.170990] RAX: 0000000000007e4a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6448.179452] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6448.184962] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6448.187924] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6448.194124] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6448.200196] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6448.211194] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6448.217338] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6448.222061] Call Trace: [ 6448.223459] [ 6448.224430] ? show_regs.cold.9+0x22/0x2f [ 6448.227028] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6448.228777] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6448.235210] ? watchdog+0x40/0x40 [ 6448.237064] ? hrtimer_interrupt+0x128/0x2c0 [ 6448.240139] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6448.242143] ? apic_timer_interrupt+0xf/0x20 [ 6448.244239] [ 6448.245823] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6448.248357] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6448.251182] do_raw_spin_lock+0x177/0x190 [ 6448.252840] _raw_spin_lock+0x21/0x30 [ 6448.254158] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6448.255866] process_one_work+0x2c8/0x700 [ 6448.256829] worker_thread+0x296/0x6e0 [ 6448.260110] ? rescuer_thread+0x570/0x570 [ 6448.261367] kthread+0x1d1/0x200 [ 6448.265355] ? set_kthread_struct+0x70/0x70 [ 6448.267021] ret_from_fork+0x1f/0x30 [ 6460.059761] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 6460.068836] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6460.106806] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6460.118857] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6460.125447] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6460.129065] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6460.140886] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6460.144411] RAX: 0000000000006c1d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6460.148009] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6460.151751] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6460.154964] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6460.158380] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6460.161571] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6460.164934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6460.169257] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6460.171862] Call Trace: [ 6460.177683] [ 6460.179013] ? show_regs.cold.9+0x22/0x2f [ 6460.182029] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6460.184690] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6460.188647] ? watchdog+0x40/0x40 [ 6460.196895] ? hrtimer_interrupt+0x128/0x2c0 [ 6460.201257] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6460.203138] ? apic_timer_interrupt+0xf/0x20 [ 6460.206419] [ 6460.207408] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6460.211828] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6460.217039] do_raw_spin_lock+0x177/0x190 [ 6460.219470] _raw_spin_lock+0x21/0x30 [ 6460.221476] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6460.225092] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6460.231548] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6460.240865] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6460.249502] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6460.253104] ? woken_wake_function+0x30/0x30 [ 6460.258256] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6460.263629] kthread+0x1d1/0x200 [ 6460.267454] ? set_kthread_struct+0x70/0x70 [ 6460.271112] ret_from_fork+0x1f/0x30 [ 6464.056025] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 6464.059052] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6464.082294] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6464.087020] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6464.090747] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6464.093563] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6464.102897] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6464.107614] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6464.109922] RDX: 0000000000007112 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6464.113279] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6464.116506] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6464.119580] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6464.123234] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6464.126565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6464.129280] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6464.133076] Call Trace: [ 6464.134103] [ 6464.135149] ? show_regs.cold.9+0x22/0x2f [ 6464.136901] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6464.139032] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6464.141194] ? watchdog+0x40/0x40 [ 6464.142214] ? hrtimer_interrupt+0x128/0x2c0 [ 6464.144007] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6464.145807] ? apic_timer_interrupt+0xf/0x20 [ 6464.147610] [ 6464.148636] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6464.151442] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6464.153631] do_raw_spin_lock+0x177/0x190 [ 6464.154833] _raw_spin_lock+0x21/0x30 [ 6464.155987] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6464.158293] obd_mod_free+0x46/0x140 [ptlrpc] [ 6464.160982] mdc_commit_open+0x91/0xb0 [mdc] [ 6464.162712] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6464.164848] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6464.167615] after_reply+0xe2c/0x1780 [ptlrpc] [ 6464.169882] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6464.172236] ? __next_timer_interrupt+0x160/0x160 [ 6464.174432] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6464.175977] ? do_wait_intr+0xf0/0xf0 [ 6464.178727] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6464.181611] kthread+0x1d1/0x200 [ 6464.183699] ? set_kthread_struct+0x70/0x70 [ 6464.185077] ret_from_fork+0x1f/0x30 [ 6470.113130] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3956s! [ 6470.118085] Showing busy workqueues and worker pools: [ 6470.123192] workqueue events: flags=0x0 [ 6470.125536] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6470.129640] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6470.135807] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6470.142305] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6470.148543] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6470.152314] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6470.157315] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6470.161315] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6470.165536] workqueue events_long: flags=0x0 [ 6470.171239] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6470.177138] pending: lu_objects_flush [obdclass] [ 6470.180033] workqueue events_unbound: flags=0x2 [ 6470.183080] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6470.186499] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6470.192960] workqueue events_power_efficient: flags=0x80 [ 6470.198807] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6470.206111] pending: neigh_periodic_work, do_cache_clean [ 6470.213113] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6470.220718] pending: neigh_periodic_work, check_lifetime [ 6470.223788] workqueue mm_percpu_wq: flags=0x8 [ 6470.225288] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6470.230571] pending: vmstat_update, lru_add_drain_per_cpu [ 6470.233358] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6470.238093] pending: vmstat_update [ 6470.242402] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6470.247127] pending: lru_add_drain_per_cpu BAR(48) [ 6470.250273] workqueue ipv6_addrconf: flags=0x40008 [ 6470.255626] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6470.262275] pending: addrconf_verify_work [ 6470.268630] workqueue knbd0-recv: flags=0x1a [ 6470.270268] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6470.278239] in-flight: 650:recv_work [ 6470.282788] workqueue ptlrpc_pinger: flags=0x2 [ 6470.287482] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6470.290981] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6470.294172] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3957s workers=4 idle: 36 466 29757 [ 6470.298357] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6460s workers=2 idle: 652 [ 6472.062679] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6472.068024] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6472.135760] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6472.149239] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6472.157047] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6472.163018] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6472.182260] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6472.187036] RAX: 0000000000007f8b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6472.192206] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6472.196184] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6472.199514] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6472.210756] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6472.215161] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6472.219142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6472.222336] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6472.226131] Call Trace: [ 6472.227501] [ 6472.228359] ? show_regs.cold.9+0x22/0x2f [ 6472.230109] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6472.233275] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6472.237112] ? watchdog+0x40/0x40 [ 6472.239294] ? hrtimer_interrupt+0x128/0x2c0 [ 6472.240828] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6472.243299] ? apic_timer_interrupt+0xf/0x20 [ 6472.245659] [ 6472.246958] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6472.251114] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6472.257626] do_raw_spin_lock+0x177/0x190 [ 6472.260331] _raw_spin_lock+0x21/0x30 [ 6472.262279] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6472.266863] ? do_raw_spin_unlock+0x75/0x190 [ 6472.269193] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6472.272270] ? do_wait_intr+0xf0/0xf0 [ 6472.274158] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6472.276958] mdc_reint+0x2d/0x110 [mdc] [ 6472.279090] mdc_unlink+0x658/0x870 [mdc] [ 6472.281015] lmv_unlink+0x34d/0xbd0 [lmv] [ 6472.283189] ll_unlink+0x24d/0x7f0 [lustre] [ 6472.285614] vfs_unlink+0x202/0x330 [ 6472.287194] do_unlinkat+0x2fa/0x460 [ 6472.288850] __x64_sys_unlink+0x37/0x50 [ 6472.290352] do_syscall_64+0xc1/0x440 [ 6472.292016] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6472.294528] RIP: 0033:0x7f83e5f5c10b [ 6472.296098] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6472.304529] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6472.314031] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6472.317910] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6472.323890] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6472.327301] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6472.331905] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6476.060062] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 6476.065208] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6476.098914] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6476.106050] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6476.109275] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6476.116185] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6476.120188] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6476.126046] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6476.128665] RAX: 000000000000450c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6476.130912] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6476.133008] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6476.135471] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6476.137027] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6476.139817] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6476.142892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6476.145093] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6476.150851] Call Trace: [ 6476.152858] [ 6476.153705] ? show_regs.cold.9+0x22/0x2f [ 6476.155190] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6476.158898] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6476.163920] ? watchdog+0x40/0x40 [ 6476.164953] ? hrtimer_interrupt+0x128/0x2c0 [ 6476.166902] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6476.168643] ? apic_timer_interrupt+0xf/0x20 [ 6476.174170] [ 6476.174813] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6476.179874] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6476.184104] do_raw_spin_lock+0x177/0x190 [ 6476.189008] _raw_spin_lock+0x21/0x30 [ 6476.194253] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6476.200098] process_one_work+0x2c8/0x700 [ 6476.201838] worker_thread+0x296/0x6e0 [ 6476.207554] ? rescuer_thread+0x570/0x570 [ 6476.212613] kthread+0x1d1/0x200 [ 6476.214118] ? set_kthread_struct+0x70/0x70 [ 6476.218648] ret_from_fork+0x1f/0x30 [ 6488.059121] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 6488.069821] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6488.105139] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6488.110482] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6488.114591] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6488.125223] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6488.146988] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6488.153822] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6488.158161] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6488.165108] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6488.169090] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6488.172544] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6488.175765] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6488.198102] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6488.204634] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6488.213924] Call Trace: [ 6488.215482] [ 6488.219168] ? show_regs.cold.9+0x22/0x2f [ 6488.223682] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6488.227794] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6488.232528] ? watchdog+0x40/0x40 [ 6488.234882] ? hrtimer_interrupt+0x128/0x2c0 [ 6488.237833] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6488.245114] ? apic_timer_interrupt+0xf/0x20 [ 6488.251086] [ 6488.252278] ? native_safe_halt+0x2e/0x50 [ 6488.256003] kvm_wait+0xcd/0xe0 [ 6488.259019] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6488.263089] do_raw_spin_lock+0x177/0x190 [ 6488.266944] _raw_spin_lock+0x21/0x30 [ 6488.272759] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6488.277556] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6488.282667] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6488.289989] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6488.293675] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6488.298830] ? woken_wake_function+0x30/0x30 [ 6488.301163] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6488.308871] kthread+0x1d1/0x200 [ 6488.313215] ? set_kthread_struct+0x70/0x70 [ 6488.316620] ret_from_fork+0x1f/0x30 [ 6496.059119] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 6496.063137] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6496.103024] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6496.109844] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6496.115283] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6496.117950] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6496.126720] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6496.130071] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6496.133213] RDX: 00000000000018fd RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6496.136532] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6496.140006] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6496.143582] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6496.147203] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6496.151022] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6496.153705] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6496.157236] Call Trace: [ 6496.158393] [ 6496.159372] ? show_regs.cold.9+0x22/0x2f [ 6496.161236] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6496.163666] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6496.165700] ? watchdog+0x40/0x40 [ 6496.167479] ? hrtimer_interrupt+0x128/0x2c0 [ 6496.169427] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6496.171563] ? apic_timer_interrupt+0xf/0x20 [ 6496.173654] [ 6496.174596] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6496.177012] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6496.179677] do_raw_spin_lock+0x177/0x190 [ 6496.181594] _raw_spin_lock+0x21/0x30 [ 6496.183518] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6496.185651] obd_mod_free+0x46/0x140 [ptlrpc] [ 6496.189354] mdc_commit_open+0x91/0xb0 [mdc] [ 6496.191640] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6496.194334] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6496.197228] after_reply+0xe2c/0x1780 [ptlrpc] [ 6496.199706] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6496.201887] ? __next_timer_interrupt+0x160/0x160 [ 6496.203995] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6496.206236] ? do_wait_intr+0xf0/0xf0 [ 6496.208587] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6496.211966] kthread+0x1d1/0x200 [ 6496.213544] ? set_kthread_struct+0x70/0x70 [ 6496.215324] ret_from_fork+0x1f/0x30 [ 6500.062122] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6500.067578] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6500.101326] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6500.108885] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6500.116350] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6500.118479] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6500.125529] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6500.130696] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6500.135972] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6500.143246] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6500.149497] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6500.158322] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6500.162366] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6500.164560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6500.167267] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6500.170321] Call Trace: [ 6500.171037] [ 6500.171865] ? show_regs.cold.9+0x22/0x2f [ 6500.173856] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6500.176457] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6500.178746] ? watchdog+0x40/0x40 [ 6500.180890] ? hrtimer_interrupt+0x128/0x2c0 [ 6500.183082] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6500.187992] ? apic_timer_interrupt+0xf/0x20 [ 6500.190450] [ 6500.191909] ? native_safe_halt+0x2e/0x50 [ 6500.194467] kvm_wait+0xcd/0xe0 [ 6500.196362] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6500.198959] do_raw_spin_lock+0x177/0x190 [ 6500.200042] _raw_spin_lock+0x21/0x30 [ 6500.201812] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6500.203701] ? do_raw_spin_unlock+0x75/0x190 [ 6500.204769] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6500.206994] ? do_wait_intr+0xf0/0xf0 [ 6500.208560] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6500.212321] mdc_reint+0x2d/0x110 [mdc] [ 6500.217725] mdc_unlink+0x658/0x870 [mdc] [ 6500.219714] lmv_unlink+0x34d/0xbd0 [lmv] [ 6500.222053] ll_unlink+0x24d/0x7f0 [lustre] [ 6500.233963] vfs_unlink+0x202/0x330 [ 6500.235458] do_unlinkat+0x2fa/0x460 [ 6500.237144] __x64_sys_unlink+0x37/0x50 [ 6500.238178] do_syscall_64+0xc1/0x440 [ 6500.239181] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6500.245565] RIP: 0033:0x7f83e5f5c10b [ 6500.248079] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6500.262960] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6500.267276] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6500.269417] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6500.273331] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6500.275719] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6500.278851] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6500.323267] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3987s! [ 6500.326819] Showing busy workqueues and worker pools: [ 6500.329448] workqueue events: flags=0x0 [ 6500.333062] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6500.337342] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6500.352639] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6500.360120] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6500.366279] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6500.370773] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6500.378702] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6500.387596] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6500.398854] workqueue events_long: flags=0x0 [ 6500.404023] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6500.408366] pending: lu_objects_flush [obdclass] [ 6500.410843] workqueue events_unbound: flags=0x2 [ 6500.419611] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6500.435669] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6500.448485] workqueue events_power_efficient: flags=0x80 [ 6500.456499] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6500.467430] pending: neigh_periodic_work, do_cache_clean [ 6500.478937] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6500.491753] pending: neigh_periodic_work, check_lifetime [ 6500.502176] workqueue mm_percpu_wq: flags=0x8 [ 6500.503874] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6500.510982] pending: vmstat_update, lru_add_drain_per_cpu [ 6500.519928] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6500.523018] pending: vmstat_update [ 6500.529053] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6500.535934] pending: lru_add_drain_per_cpu BAR(48) [ 6500.543857] workqueue ipv6_addrconf: flags=0x40008 [ 6500.556132] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6500.562750] pending: addrconf_verify_work [ 6500.565486] workqueue knbd0-recv: flags=0x1a [ 6500.569046] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6500.572956] in-flight: 650:recv_work [ 6500.575465] workqueue ptlrpc_pinger: flags=0x2 [ 6500.577756] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6500.582252] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6500.591275] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3987s workers=4 idle: 36 466 29757 [ 6500.601288] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6490s workers=2 idle: 652 [ 6504.060547] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6504.064576] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6504.081238] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6504.089719] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6504.103331] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6504.110882] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6504.116608] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6504.131624] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6504.137790] RAX: 0000000000007810 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6504.144733] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6504.148470] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6504.155054] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6504.161530] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6504.167523] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6504.173700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6504.177627] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6504.184383] Call Trace: [ 6504.186573] [ 6504.188495] ? show_regs.cold.9+0x22/0x2f [ 6504.191804] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6504.193733] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6504.199428] ? watchdog+0x40/0x40 [ 6504.202037] ? hrtimer_interrupt+0x128/0x2c0 [ 6504.205011] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6504.210332] ? apic_timer_interrupt+0xf/0x20 [ 6504.215050] [ 6504.216514] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6504.222236] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6504.224648] do_raw_spin_lock+0x177/0x190 [ 6504.230152] _raw_spin_lock+0x21/0x30 [ 6504.233692] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6504.238172] process_one_work+0x2c8/0x700 [ 6504.240674] worker_thread+0x296/0x6e0 [ 6504.244837] ? rescuer_thread+0x570/0x570 [ 6504.246592] kthread+0x1d1/0x200 [ 6504.251762] ? set_kthread_struct+0x70/0x70 [ 6504.257333] ret_from_fork+0x1f/0x30 [ 6516.059065] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 6516.065723] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6516.090307] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6516.096397] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6516.101266] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6516.104959] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6516.114770] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6516.120327] RAX: 0000000000002726 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6516.125867] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6516.129497] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6516.133321] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6516.137447] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6516.141492] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6516.145610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6516.148531] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6516.151330] Call Trace: [ 6516.152401] [ 6516.153438] ? show_regs.cold.9+0x22/0x2f [ 6516.155261] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6516.157108] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6516.159208] ? watchdog+0x40/0x40 [ 6516.160454] ? hrtimer_interrupt+0x128/0x2c0 [ 6516.162074] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6516.164539] ? apic_timer_interrupt+0xf/0x20 [ 6516.166508] [ 6516.167569] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6516.170568] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6516.175498] do_raw_spin_lock+0x177/0x190 [ 6516.179773] _raw_spin_lock+0x21/0x30 [ 6516.182554] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6516.185380] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6516.192179] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6516.198139] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6516.201278] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6516.204491] ? woken_wake_function+0x30/0x30 [ 6516.207348] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6516.213015] kthread+0x1d1/0x200 [ 6516.215230] ? set_kthread_struct+0x70/0x70 [ 6516.218362] ret_from_fork+0x1f/0x30 [ 6528.056063] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 6528.059359] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6528.062391] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6528.075524] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6528.075540] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6528.075544] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6528.075557] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6528.075562] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6528.075568] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6528.075571] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6528.075573] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6528.075576] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6528.075579] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6528.075586] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6528.075589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6528.075592] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6528.075601] Call Trace: [ 6528.075606] [ 6528.075613] ? show_regs.cold.9+0x22/0x2f [ 6528.075637] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6528.075643] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6528.075648] ? watchdog+0x40/0x40 [ 6528.075652] ? hrtimer_interrupt+0x128/0x2c0 [ 6528.075657] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6528.075664] ? apic_timer_interrupt+0xf/0x20 [ 6528.075669] [ 6528.075672] ? native_safe_halt+0x2e/0x50 [ 6528.075676] kvm_wait+0xcd/0xe0 [ 6528.075684] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6528.075692] do_raw_spin_lock+0x177/0x190 [ 6528.075697] _raw_spin_lock+0x21/0x30 [ 6528.075704] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6528.076105] obd_mod_free+0x46/0x140 [ptlrpc] [ 6528.079176] Modules linked in: [ 6528.082334] mdc_commit_open+0x91/0xb0 [mdc] [ 6528.082387] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6528.082757] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6528.083175] after_reply+0xe2c/0x1780 [ptlrpc] [ 6528.083390] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6528.083611] ? __next_timer_interrupt+0x160/0x160 [ 6528.083635] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6528.083871] ? do_wait_intr+0xf0/0xf0 [ 6528.083880] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6528.088087] lustre(O) [ 6528.090056] kthread+0x1d1/0x200 [ 6528.090073] ? set_kthread_struct+0x70/0x70 [ 6528.090078] ret_from_fork+0x1f/0x30 [ 6528.222589] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6528.251174] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6528.258403] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6528.264190] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6528.265494] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6528.280129] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6528.284259] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6528.288797] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6528.293150] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6528.297409] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6528.303539] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6528.309955] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6528.312442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6528.317229] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6528.320312] Call Trace: [ 6528.323196] [ 6528.324142] ? show_regs.cold.9+0x22/0x2f [ 6528.326367] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6528.330154] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6528.331727] ? watchdog+0x40/0x40 [ 6528.335990] ? hrtimer_interrupt+0x128/0x2c0 [ 6528.337264] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6528.340273] ? apic_timer_interrupt+0xf/0x20 [ 6528.343930] [ 6528.344508] ? native_safe_halt+0x2e/0x50 [ 6528.347261] kvm_wait+0xcd/0xe0 [ 6528.350197] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6528.352220] do_raw_spin_lock+0x177/0x190 [ 6528.357222] _raw_spin_lock+0x21/0x30 [ 6528.359369] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6528.363423] ? do_raw_spin_unlock+0x75/0x190 [ 6528.366985] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6528.370801] ? do_wait_intr+0xf0/0xf0 [ 6528.372817] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6528.377695] mdc_reint+0x2d/0x110 [mdc] [ 6528.378954] mdc_unlink+0x658/0x870 [mdc] [ 6528.380240] lmv_unlink+0x34d/0xbd0 [lmv] [ 6528.384260] ll_unlink+0x24d/0x7f0 [lustre] [ 6528.386356] vfs_unlink+0x202/0x330 [ 6528.387392] do_unlinkat+0x2fa/0x460 [ 6528.394124] __x64_sys_unlink+0x37/0x50 [ 6528.395291] do_syscall_64+0xc1/0x440 [ 6528.400363] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6528.404366] RIP: 0033:0x7f83e5f5c10b [ 6528.405470] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6528.415989] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6528.421129] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6528.426204] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6528.430416] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6528.436195] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6528.441219] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6531.041053] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4017s! [ 6531.043786] Showing busy workqueues and worker pools: [ 6531.045900] workqueue events: flags=0x0 [ 6531.047059] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6531.049603] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6531.052923] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6531.055355] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6531.057701] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6531.060274] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6531.063063] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6531.065368] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6531.069063] workqueue events_long: flags=0x0 [ 6531.070695] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6531.073148] pending: lu_objects_flush [obdclass] [ 6531.075764] workqueue events_unbound: flags=0x2 [ 6531.077375] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6531.079844] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6531.082367] workqueue events_power_efficient: flags=0x80 [ 6531.084441] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6531.091209] pending: neigh_periodic_work, do_cache_clean [ 6531.094346] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6531.098141] pending: neigh_periodic_work, check_lifetime [ 6531.100415] workqueue mm_percpu_wq: flags=0x8 [ 6531.102633] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6531.105181] pending: vmstat_update, lru_add_drain_per_cpu [ 6531.107926] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6531.111319] pending: vmstat_update [ 6531.113090] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6531.115990] pending: lru_add_drain_per_cpu BAR(48) [ 6531.118204] workqueue ipv6_addrconf: flags=0x40008 [ 6531.121379] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6531.123991] pending: addrconf_verify_work [ 6531.125301] workqueue knbd0-recv: flags=0x1a [ 6531.126455] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6531.129141] in-flight: 650:recv_work [ 6531.130702] workqueue ptlrpc_pinger: flags=0x2 [ 6531.132498] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6531.134701] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6531.137302] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4017s workers=4 idle: 36 466 29757 [ 6531.140653] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6521s workers=2 idle: 652 [ 6532.060994] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6532.067459] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6532.091696] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6532.100921] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6532.106775] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6532.111044] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6532.113905] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6532.123760] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6532.128320] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6532.134441] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6532.137512] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6532.141378] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6532.144344] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6532.147910] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6532.152682] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6532.154655] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6532.158090] Call Trace: [ 6532.159328] [ 6532.160895] ? show_regs.cold.9+0x22/0x2f [ 6532.162917] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6532.164960] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6532.167072] ? watchdog+0x40/0x40 [ 6532.170121] ? hrtimer_interrupt+0x128/0x2c0 [ 6532.172258] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6532.177140] ? apic_timer_interrupt+0xf/0x20 [ 6532.179406] [ 6532.180398] ? native_safe_halt+0x2e/0x50 [ 6532.181965] kvm_wait+0xcd/0xe0 [ 6532.183157] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6532.185973] do_raw_spin_lock+0x177/0x190 [ 6532.187118] _raw_spin_lock+0x21/0x30 [ 6532.188114] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6532.193689] process_one_work+0x2c8/0x700 [ 6532.195486] worker_thread+0x296/0x6e0 [ 6532.197690] ? rescuer_thread+0x570/0x570 [ 6532.199515] kthread+0x1d1/0x200 [ 6532.201066] ? set_kthread_struct+0x70/0x70 [ 6532.202564] ret_from_fork+0x1f/0x30 [ 6541.064469] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 6541.075100] rcu: 0-....: (2906203 ticks this GP) idle=30a/1/0x4000000000000000 softirq=128519/128519 fqs=126395 [ 6541.085592] (detected by 1, t=4560105 jiffies, g=262441, q=58488) [ 6541.092054] Sending NMI from CPU 1 to CPUs 0: [ 6541.099328] NMI backtrace for cpu 0 [ 6541.099336] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6541.099339] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6541.099341] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6541.099343] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6541.099346] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 [ 6541.099348] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6541.099350] RDX: 0000000000000875 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6541.099352] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6541.099354] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6541.099355] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6541.099357] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6541.099365] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6541.099367] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6541.099368] Call Trace: [ 6541.099369] [ 6541.099370] ? show_regs.cold.9+0x22/0x2f [ 6541.099372] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6541.099373] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6541.099374] ? nmi_handle+0xbd/0x210 [ 6541.099375] ? default_do_nmi+0x4b/0x1b0 [ 6541.099376] ? do_nmi+0x280/0x380 [ 6541.099377] ? end_repeat_nmi+0x16/0x1f [ 6541.099378] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6541.099380] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6541.099381] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6541.099383] [ 6541.099384] do_raw_spin_lock+0x177/0x190 [ 6541.099385] _raw_spin_lock+0x21/0x30 [ 6541.099386] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6541.099387] obd_mod_free+0x46/0x140 [ptlrpc] [ 6541.099388] mdc_commit_open+0x91/0xb0 [mdc] [ 6541.099389] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6541.099391] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6541.099392] after_reply+0xe2c/0x1780 [ptlrpc] [ 6541.099394] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6541.099395] ? __next_timer_interrupt+0x160/0x160 [ 6541.099396] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6541.099397] ? do_wait_intr+0xf0/0xf0 [ 6541.099399] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6541.099400] kthread+0x1d1/0x200 [ 6541.099401] ? set_kthread_struct+0x70/0x70 [ 6541.099402] ret_from_fork+0x1f/0x30 [ 6541.101265] rcu: rcu_sched kthread starved for 4029292 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 6541.285719] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6541.292284] rcu: RCU grace-period kthread stack dump: [ 6541.296871] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6541.301922] Call Trace: [ 6541.303318] __schedule+0x351/0xcb0 [ 6541.305255] schedule+0xc0/0x180 [ 6541.306917] schedule_timeout+0xb4/0x190 [ 6541.308853] ? do_raw_spin_unlock+0x75/0x190 [ 6541.310924] ? __next_timer_interrupt+0x160/0x160 [ 6541.314028] ? prepare_to_swait_event+0xf3/0x140 [ 6541.316485] rcu_gp_kthread+0x8dd/0x15b0 [ 6541.318379] ? force_qs_rnp+0x2c0/0x2c0 [ 6541.320669] kthread+0x1d1/0x200 [ 6541.322235] ? set_kthread_struct+0x70/0x70 [ 6541.323988] ret_from_fork+0x1f/0x30 [ 6541.325651] rcu: Stack dump where RCU GP kthread last ran: [ 6541.327784] Sending NMI from CPU 1 to CPUs 3: [ 6541.331505] NMI backtrace for cpu 3 [ 6541.331515] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6541.331518] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6541.331520] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6541.331521] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6541.331524] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 [ 6541.331526] RAX: 0000000000003185 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6541.331528] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6541.331529] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6541.331531] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6541.331532] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6541.331533] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6541.331534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6541.331535] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6541.331536] Call Trace: [ 6541.331537] [ 6541.331538] ? show_regs.cold.9+0x22/0x2f [ 6541.331539] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6541.331540] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6541.331541] ? nmi_handle+0xbd/0x210 [ 6541.331542] ? default_do_nmi+0x4b/0x1b0 [ 6541.331543] ? do_nmi+0x280/0x380 [ 6541.331544] ? end_repeat_nmi+0x16/0x1f [ 6541.331545] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6541.331547] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6541.331548] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6541.331549] [ 6541.331549] do_raw_spin_lock+0x177/0x190 [ 6541.331550] _raw_spin_lock+0x21/0x30 [ 6541.331551] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6541.331553] ? do_raw_spin_unlock+0x75/0x190 [ 6541.331554] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6541.331555] ? do_wait_intr+0xf0/0xf0 [ 6541.331556] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6541.331558] mdc_reint+0x2d/0x110 [mdc] [ 6541.331559] mdc_unlink+0x658/0x870 [mdc] [ 6541.331560] lmv_unlink+0x34d/0xbd0 [lmv] [ 6541.331561] ll_unlink+0x24d/0x7f0 [lustre] [ 6541.331562] vfs_unlink+0x202/0x330 [ 6541.331563] do_unlinkat+0x2fa/0x460 [ 6541.331565] __x64_sys_unlink+0x37/0x50 [ 6541.331565] do_syscall_64+0xc1/0x440 [ 6541.331567] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6541.331568] RIP: 0033:0x7f83e5f5c10b [ 6541.331570] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6541.331572] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6541.331574] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6541.331576] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6541.331577] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6541.331578] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6541.331580] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6560.066198] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6560.073032] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6560.118267] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6560.127001] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6560.130868] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6560.133080] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6560.139525] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6560.152589] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6560.156004] RAX: 0000000000002c82 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6560.159909] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6560.164403] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6560.167813] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6560.172209] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6560.175508] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6560.179241] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6560.181884] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6560.185610] Call Trace: [ 6560.186887] [ 6560.187878] ? show_regs.cold.9+0x22/0x2f [ 6560.189848] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6560.192396] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6560.194618] ? watchdog+0x40/0x40 [ 6560.195955] ? hrtimer_interrupt+0x128/0x2c0 [ 6560.197909] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6560.202675] ? apic_timer_interrupt+0xf/0x20 [ 6560.206351] [ 6560.207814] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6560.212239] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6560.217242] do_raw_spin_lock+0x177/0x190 [ 6560.223892] _raw_spin_lock+0x21/0x30 [ 6560.225872] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6560.231937] process_one_work+0x2c8/0x700 [ 6560.236080] worker_thread+0x296/0x6e0 [ 6560.241459] ? rescuer_thread+0x570/0x570 [ 6560.246125] kthread+0x1d1/0x200 [ 6560.249777] ? set_kthread_struct+0x70/0x70 [ 6560.256030] ret_from_fork+0x1f/0x30 [ 6561.248104] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4048s! [ 6561.251137] Showing busy workqueues and worker pools: [ 6561.253406] workqueue events: flags=0x0 [ 6561.254933] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6561.257546] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6561.260997] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6561.263719] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6561.266313] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6561.268740] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6561.272398] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6561.275038] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6561.278584] workqueue events_long: flags=0x0 [ 6561.280158] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6561.283504] pending: lu_objects_flush [obdclass] [ 6561.285810] workqueue events_unbound: flags=0x2 [ 6561.288631] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6561.291244] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6561.294551] workqueue events_power_efficient: flags=0x80 [ 6561.297099] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6561.300510] pending: neigh_periodic_work, do_cache_clean [ 6561.303258] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6561.306133] pending: neigh_periodic_work, check_lifetime [ 6561.308549] workqueue mm_percpu_wq: flags=0x8 [ 6561.310211] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6561.313153] pending: vmstat_update, lru_add_drain_per_cpu [ 6561.315483] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6561.318826] pending: vmstat_update [ 6561.320476] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6561.323440] pending: lru_add_drain_per_cpu BAR(48) [ 6561.325618] workqueue ipv6_addrconf: flags=0x40008 [ 6561.327678] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6561.330858] pending: addrconf_verify_work [ 6561.332151] workqueue knbd0-recv: flags=0x1a [ 6561.333467] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6561.335573] in-flight: 650:recv_work [ 6561.336748] workqueue ptlrpc_pinger: flags=0x2 [ 6561.338457] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6561.340868] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6561.342905] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4048s workers=4 idle: 36 466 29757 [ 6561.345737] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6551s workers=2 idle: 652 [ 6568.061498] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 6568.062023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6568.069865] Modules linked in: [ 6568.072796] Modules linked in: [ 6568.072797] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) [ 6568.073926] lustre(O) [ 6568.078537] mgs(O) [ 6568.080406] osp(O) [ 6568.081377] lquota(O) [ 6568.081988] ofd(O) [ 6568.082532] lfsck(O) [ 6568.083045] lod(O) [ 6568.083831] obdecho(O) [ 6568.084849] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6568.084915] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6568.084923] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6568.084925] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6568.084937] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6568.085042] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6568.085047] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6568.085049] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6568.085051] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6568.085053] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6568.085056] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6568.085059] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6568.085062] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6568.085064] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6568.085072] Call Trace: [ 6568.085078] [ 6568.085084] ? show_regs.cold.9+0x22/0x2f [ 6568.085094] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6568.085167] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6568.085172] ? watchdog+0x40/0x40 [ 6568.085176] ? hrtimer_interrupt+0x128/0x2c0 [ 6568.085180] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6568.085186] ? apic_timer_interrupt+0xf/0x20 [ 6568.085191] [ 6568.085193] ? native_safe_halt+0x2e/0x50 [ 6568.085197] kvm_wait+0xcd/0xe0 [ 6568.085203] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6568.085210] do_raw_spin_lock+0x177/0x190 [ 6568.085214] _raw_spin_lock+0x21/0x30 [ 6568.085220] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6568.085621] ? do_raw_spin_unlock+0x75/0x190 [ 6568.085627] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6568.085840] ? do_wait_intr+0xf0/0xf0 [ 6568.085848] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6568.086346] mdc_reint+0x2d/0x110 [mdc] [ 6568.086391] mdc_unlink+0x658/0x870 [mdc] [ 6568.086417] lmv_unlink+0x34d/0xbd0 [lmv] [ 6568.086457] ll_unlink+0x24d/0x7f0 [lustre] [ 6568.086699] vfs_unlink+0x202/0x330 [ 6568.086708] do_unlinkat+0x2fa/0x460 [ 6568.086713] __x64_sys_unlink+0x37/0x50 [ 6568.086716] do_syscall_64+0xc1/0x440 [ 6568.086724] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6568.086729] RIP: 0033:0x7f83e5f5c10b [ 6568.086737] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6568.086741] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6568.086746] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6568.086749] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6568.086751] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6568.086753] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6568.086756] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6568.307505] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6568.322371] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6568.326711] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6568.331497] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6568.333696] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6568.342341] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6568.347468] RAX: 0000000000007d89 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6568.350247] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6568.353262] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6568.357054] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6568.359853] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6568.362881] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6568.367478] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6568.372888] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6568.375360] Call Trace: [ 6568.377362] [ 6568.378797] ? show_regs.cold.9+0x22/0x2f [ 6568.380566] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6568.382121] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6568.383903] ? watchdog+0x40/0x40 [ 6568.385337] ? hrtimer_interrupt+0x128/0x2c0 [ 6568.386758] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6568.388911] ? apic_timer_interrupt+0xf/0x20 [ 6568.390536] [ 6568.391282] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6568.394035] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6568.395876] do_raw_spin_lock+0x177/0x190 [ 6568.405747] _raw_spin_lock+0x21/0x30 [ 6568.407268] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6568.410493] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6568.413029] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6568.415394] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6568.417592] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6568.423940] ? woken_wake_function+0x30/0x30 [ 6568.426560] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6568.430200] kthread+0x1d1/0x200 [ 6568.431141] ? set_kthread_struct+0x70/0x70 [ 6568.432793] ret_from_fork+0x1f/0x30 [ 6588.058002] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 6588.060565] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6588.066741] Modules linked in: lustre(O) osp(O) ofd(O) [ 6588.070717] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6588.070942] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6588.070949] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6588.070955] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6588.072508] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6588.072521] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6588.072525] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6588.072529] RAX: 0000000000006b74 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6588.072532] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6588.072534] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6588.072536] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6588.072538] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6588.072541] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6588.072543] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6588.072546] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6588.072553] Call Trace: [ 6588.072559] [ 6588.072564] ? show_regs.cold.9+0x22/0x2f [ 6588.072573] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6588.072579] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6588.072583] ? watchdog+0x40/0x40 [ 6588.073818] ? hrtimer_interrupt+0x128/0x2c0 [ 6588.073825] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6588.073834] ? apic_timer_interrupt+0xf/0x20 [ 6588.073839] [ 6588.073843] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6588.073849] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6588.073860] do_raw_spin_lock+0x177/0x190 [ 6588.073865] _raw_spin_lock+0x21/0x30 [ 6588.073870] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6588.074429] process_one_work+0x2c8/0x700 [ 6588.074436] worker_thread+0x296/0x6e0 [ 6588.074440] ? rescuer_thread+0x570/0x570 [ 6588.074443] kthread+0x1d1/0x200 [ 6588.074448] ? set_kthread_struct+0x70/0x70 [ 6588.074451] ret_from_fork+0x1f/0x30 [ 6588.293747] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6588.342871] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6588.352774] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6588.358734] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6588.365781] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6588.390162] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6588.398508] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6588.409264] RDX: 0000000000001a1f RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6588.419023] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6588.425892] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6588.435741] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6588.441644] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6588.448713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6588.457829] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6588.469322] Call Trace: [ 6588.475605] [ 6588.478117] ? show_regs.cold.9+0x22/0x2f [ 6588.484692] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6588.486709] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6588.495053] ? watchdog+0x40/0x40 [ 6588.497154] ? hrtimer_interrupt+0x128/0x2c0 [ 6588.501743] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6588.508141] ? apic_timer_interrupt+0xf/0x20 [ 6588.515105] [ 6588.517173] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6588.528082] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6588.530727] do_raw_spin_lock+0x177/0x190 [ 6588.533741] _raw_spin_lock+0x21/0x30 [ 6588.537679] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6588.542066] obd_mod_free+0x46/0x140 [ptlrpc] [ 6588.548840] mdc_commit_open+0x91/0xb0 [mdc] [ 6588.552484] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6588.559022] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6588.570874] after_reply+0xe2c/0x1780 [ptlrpc] [ 6588.577715] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6588.586964] ? __next_timer_interrupt+0x160/0x160 [ 6588.591522] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6588.593336] ? do_wait_intr+0xf0/0xf0 [ 6588.598691] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6588.605846] kthread+0x1d1/0x200 [ 6588.610110] ? set_kthread_struct+0x70/0x70 [ 6588.615122] ret_from_fork+0x1f/0x30 [ 6591.458105] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4078s! [ 6591.461312] Showing busy workqueues and worker pools: [ 6591.476103] workqueue events: flags=0x0 [ 6591.484081] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6591.498631] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6591.513532] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6591.517308] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6591.527149] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6591.529700] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6591.545407] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6591.555430] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6591.562570] workqueue events_long: flags=0x0 [ 6591.569636] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6591.574230] pending: lu_objects_flush [obdclass] [ 6591.579509] workqueue events_unbound: flags=0x2 [ 6591.587987] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6591.591501] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6591.596507] workqueue events_power_efficient: flags=0x80 [ 6591.606819] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6591.614883] pending: neigh_periodic_work, do_cache_clean [ 6591.618614] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6591.622233] pending: neigh_periodic_work, check_lifetime [ 6591.626869] workqueue mm_percpu_wq: flags=0x8 [ 6591.632483] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6591.639066] pending: vmstat_update, lru_add_drain_per_cpu [ 6591.642467] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6591.649501] pending: vmstat_update [ 6591.653976] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6591.662585] pending: lru_add_drain_per_cpu BAR(48) [ 6591.666231] workqueue ipv6_addrconf: flags=0x40008 [ 6591.667979] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6591.678902] pending: addrconf_verify_work [ 6591.681631] workqueue knbd0-recv: flags=0x1a [ 6591.687235] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6591.691659] in-flight: 650:recv_work [ 6591.697392] workqueue ptlrpc_pinger: flags=0x2 [ 6591.701892] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6591.706960] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6591.709961] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4078s workers=4 idle: 36 466 29757 [ 6591.723616] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6581s workers=2 idle: 652 [ 6596.061521] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 6596.062748] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 6596.080587] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6596.080673] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6596.080679] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6596.080682] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6596.080696] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6596.080701] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6596.080705] RAX: 00000000000038ad RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6596.080708] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6596.080710] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6596.080713] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6596.080715] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6596.080719] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6596.080722] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6596.080725] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6596.080733] Call Trace: [ 6596.080738] [ 6596.080744] ? show_regs.cold.9+0x22/0x2f [ 6596.080755] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6596.080761] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6596.080765] ? watchdog+0x40/0x40 [ 6596.080770] ? hrtimer_interrupt+0x128/0x2c0 [ 6596.080774] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6596.080782] ? apic_timer_interrupt+0xf/0x20 [ 6596.080787] [ 6596.080790] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6596.080794] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6596.080800] do_raw_spin_lock+0x177/0x190 [ 6596.080804] _raw_spin_lock+0x21/0x30 [ 6596.080809] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6596.081108] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6596.081948] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6596.083014] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6596.083907] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6596.084134] ? woken_wake_function+0x30/0x30 [ 6596.084144] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6596.092522] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6596.092604] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6596.092905] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6596.092908] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6596.092922] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6596.092926] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6596.092931] RAX: 0000000000005877 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6596.092933] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6596.092935] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6596.092938] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6596.092940] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6596.092944] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6596.092947] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6596.092949] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6596.092969] Call Trace: [ 6596.092975] [ 6596.092981] ? show_regs.cold.9+0x22/0x2f [ 6596.092991] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6596.092997] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6596.093002] ? watchdog+0x40/0x40 [ 6596.093006] ? hrtimer_interrupt+0x128/0x2c0 [ 6596.093011] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6596.093020] ? apic_timer_interrupt+0xf/0x20 [ 6596.093026] [ 6596.093029] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6596.093039] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6596.093045] do_raw_spin_lock+0x177/0x190 [ 6596.093050] _raw_spin_lock+0x21/0x30 [ 6596.093056] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6596.093600] ? do_raw_spin_unlock+0x75/0x190 [ 6596.093710] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6596.094035] ? do_wait_intr+0xf0/0xf0 [ 6596.095372] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6596.096628] mdc_reint+0x2d/0x110 [mdc] [ 6596.096675] mdc_unlink+0x658/0x870 [mdc] [ 6596.096701] lmv_unlink+0x34d/0xbd0 [lmv] [ 6596.096733] ll_unlink+0x24d/0x7f0 [lustre] [ 6596.176124] kthread+0x1d1/0x200 [ 6596.176706] ? set_kthread_struct+0x70/0x70 [ 6596.176915] ret_from_fork+0x1f/0x30 [ 6596.715877] vfs_unlink+0x202/0x330 [ 6596.717662] do_unlinkat+0x2fa/0x460 [ 6596.719725] __x64_sys_unlink+0x37/0x50 [ 6596.724444] do_syscall_64+0xc1/0x440 [ 6596.736535] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6596.742023] RIP: 0033:0x7f83e5f5c10b [ 6596.751344] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6596.779854] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6596.785438] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6596.794552] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6596.809785] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6596.826340] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6596.839689] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6616.058806] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 6616.061457] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6616.065859] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6616.065940] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6616.072185] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6616.072268] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6616.072276] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6616.072284] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6616.072814] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6616.072825] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6616.072830] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6616.072924] RAX: 0000000000007e92 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6616.072927] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6616.072930] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6616.072932] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6616.072934] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6616.072938] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6616.072941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6616.072943] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6616.072950] Call Trace: [ 6616.072957] [ 6616.072962] ? show_regs.cold.9+0x22/0x2f [ 6616.072984] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6616.072990] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6616.072994] ? watchdog+0x40/0x40 [ 6616.072999] ? hrtimer_interrupt+0x128/0x2c0 [ 6616.073004] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6616.073012] ? apic_timer_interrupt+0xf/0x20 [ 6616.073016] [ 6616.073019] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6616.073023] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6616.073028] do_raw_spin_lock+0x177/0x190 [ 6616.073033] _raw_spin_lock+0x21/0x30 [ 6616.073037] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6616.073248] process_one_work+0x2c8/0x700 [ 6616.073255] worker_thread+0x296/0x6e0 [ 6616.073259] ? rescuer_thread+0x570/0x570 [ 6616.073262] kthread+0x1d1/0x200 [ 6616.073266] ? set_kthread_struct+0x70/0x70 [ 6616.073270] ret_from_fork+0x1f/0x30 [ 6616.336626] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6616.344721] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6616.354062] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6616.369902] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6616.378322] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6616.381657] RDX: 0000000000007c1d RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6616.384798] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6616.388662] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6616.393049] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6616.396638] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6616.403710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6616.408934] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6616.414397] Call Trace: [ 6616.415957] [ 6616.417360] ? show_regs.cold.9+0x22/0x2f [ 6616.418788] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6616.421418] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6616.422993] ? watchdog+0x40/0x40 [ 6616.425108] ? hrtimer_interrupt+0x128/0x2c0 [ 6616.428251] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6616.430995] ? apic_timer_interrupt+0xf/0x20 [ 6616.432930] [ 6616.433792] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6616.439067] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6616.442609] do_raw_spin_lock+0x177/0x190 [ 6616.445344] _raw_spin_lock+0x21/0x30 [ 6616.448085] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6616.454068] obd_mod_free+0x46/0x140 [ptlrpc] [ 6616.456385] mdc_commit_open+0x91/0xb0 [mdc] [ 6616.460328] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6616.462753] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6616.465858] after_reply+0xe2c/0x1780 [ptlrpc] [ 6616.470863] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6616.477606] ? __next_timer_interrupt+0x160/0x160 [ 6616.479239] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6616.485511] ? do_wait_intr+0xf0/0xf0 [ 6616.487520] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6616.491001] kthread+0x1d1/0x200 [ 6616.492654] ? set_kthread_struct+0x70/0x70 [ 6616.495824] ret_from_fork+0x1f/0x30 [ 6622.177048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4108s! [ 6622.179837] Showing busy workqueues and worker pools: [ 6622.181458] workqueue events: flags=0x0 [ 6622.182728] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6622.185065] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6622.188305] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6622.191029] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6622.193349] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6622.195642] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6622.198278] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6622.200597] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6622.203616] workqueue events_long: flags=0x0 [ 6622.205077] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6622.207181] pending: lu_objects_flush [obdclass] [ 6622.209199] workqueue events_unbound: flags=0x2 [ 6622.210488] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6622.212822] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6622.214765] workqueue events_power_efficient: flags=0x80 [ 6622.216641] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6622.219053] pending: neigh_periodic_work, do_cache_clean [ 6622.220992] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6622.223062] pending: neigh_periodic_work, check_lifetime [ 6622.224846] workqueue mm_percpu_wq: flags=0x8 [ 6622.226394] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6622.228673] pending: vmstat_update, lru_add_drain_per_cpu [ 6622.230667] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6622.233285] pending: vmstat_update [ 6622.234846] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6622.237381] pending: lru_add_drain_per_cpu BAR(48) [ 6622.239238] workqueue ipv6_addrconf: flags=0x40008 [ 6622.240993] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6622.243329] pending: addrconf_verify_work [ 6622.244589] workqueue knbd0-recv: flags=0x1a [ 6622.246108] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6622.248419] in-flight: 650:recv_work [ 6622.249693] workqueue ptlrpc_pinger: flags=0x2 [ 6622.251703] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6622.253852] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6622.256332] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4109s workers=4 idle: 36 466 29757 [ 6622.258829] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6612s workers=2 idle: 652 [ 6624.059386] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 6624.062114] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6624.065325] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 6624.124519] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6624.140384] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6624.140464] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6624.140472] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6624.140476] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6624.140490] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6624.140494] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6624.140499] RAX: 0000000000007b10 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6624.140501] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6624.140504] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6624.140506] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6624.140508] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6624.140511] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6624.140514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6624.140516] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6624.140525] Call Trace: [ 6624.140531] [ 6624.140549] ? show_regs.cold.9+0x22/0x2f [ 6624.140560] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6624.140566] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6624.140570] ? watchdog+0x40/0x40 [ 6624.140575] ? hrtimer_interrupt+0x128/0x2c0 [ 6624.140579] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6624.140588] ? apic_timer_interrupt+0xf/0x20 [ 6624.140593] [ 6624.140596] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6624.140601] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6624.140606] do_raw_spin_lock+0x177/0x190 [ 6624.140610] _raw_spin_lock+0x21/0x30 [ 6624.140616] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6624.141726] ? do_raw_spin_unlock+0x75/0x190 [ 6624.141741] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6624.142157] ? do_wait_intr+0xf0/0xf0 [ 6624.142166] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6624.142367] mdc_reint+0x2d/0x110 [mdc] [ 6624.142407] mdc_unlink+0x658/0x870 [mdc] [ 6624.142434] lmv_unlink+0x34d/0xbd0 [lmv] [ 6624.142462] ll_unlink+0x24d/0x7f0 [lustre] [ 6624.144124] vfs_unlink+0x202/0x330 [ 6624.144134] do_unlinkat+0x2fa/0x460 [ 6624.144140] __x64_sys_unlink+0x37/0x50 [ 6624.144144] do_syscall_64+0xc1/0x440 [ 6624.144151] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6624.144160] RIP: 0033:0x7f83e5f5c10b [ 6624.144168] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6624.144172] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6624.144178] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6624.144181] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6624.144183] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6624.144185] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6624.144188] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6624.526502] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6624.536651] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6624.542502] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6624.552848] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6624.557365] RAX: 000000000000082e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6624.566513] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6624.578875] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6624.600216] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6624.611729] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6624.627644] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6624.641176] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6624.650450] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6624.657258] Call Trace: [ 6624.658147] [ 6624.662520] ? show_regs.cold.9+0x22/0x2f [ 6624.664562] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6624.672540] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6624.677195] ? watchdog+0x40/0x40 [ 6624.688034] ? hrtimer_interrupt+0x128/0x2c0 [ 6624.695415] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6624.700666] ? apic_timer_interrupt+0xf/0x20 [ 6624.714557] [ 6624.716479] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6624.725468] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6624.736324] do_raw_spin_lock+0x177/0x190 [ 6624.745866] _raw_spin_lock+0x21/0x30 [ 6624.757070] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6624.770165] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6624.779132] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6624.786478] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6624.795013] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6624.809203] ? woken_wake_function+0x30/0x30 [ 6624.815167] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6624.833995] kthread+0x1d1/0x200 [ 6624.835105] ? set_kthread_struct+0x70/0x70 [ 6624.836587] ret_from_fork+0x1f/0x30 [ 6644.062568] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6644.068265] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6644.139423] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6644.158315] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6644.164654] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6644.175265] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6644.184141] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6644.203279] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6644.208126] RAX: 0000000000000a85 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6644.213963] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6644.224179] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6644.231161] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6644.249602] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6644.261131] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6644.278952] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6644.285616] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6644.299897] Call Trace: [ 6644.306756] [ 6644.307546] ? show_regs.cold.9+0x22/0x2f [ 6644.312430] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6644.319955] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6644.326657] ? watchdog+0x40/0x40 [ 6644.329599] ? hrtimer_interrupt+0x128/0x2c0 [ 6644.334710] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6644.338959] ? apic_timer_interrupt+0xf/0x20 [ 6644.346263] [ 6644.347975] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6644.352855] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6644.356659] do_raw_spin_lock+0x177/0x190 [ 6644.360939] _raw_spin_lock+0x21/0x30 [ 6644.367973] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6644.382368] process_one_work+0x2c8/0x700 [ 6644.393967] worker_thread+0x296/0x6e0 [ 6644.406124] ? rescuer_thread+0x570/0x570 [ 6644.407722] kthread+0x1d1/0x200 [ 6644.419486] ? set_kthread_struct+0x70/0x70 [ 6644.425597] ret_from_fork+0x1f/0x30 [ 6648.056159] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 6648.065268] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6648.098064] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6648.106696] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6648.111021] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6648.116730] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6648.132598] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6648.138935] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6648.144802] RDX: 00000000000050b5 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6648.151581] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6648.156684] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6648.160945] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6648.167753] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6648.172793] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6648.179554] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6648.189748] Call Trace: [ 6648.191070] [ 6648.192702] ? show_regs.cold.9+0x22/0x2f [ 6648.196058] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6648.201888] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6648.204806] ? watchdog+0x40/0x40 [ 6648.207480] ? hrtimer_interrupt+0x128/0x2c0 [ 6648.209913] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6648.212577] ? apic_timer_interrupt+0xf/0x20 [ 6648.217228] [ 6648.218841] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6648.222182] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6648.225003] do_raw_spin_lock+0x177/0x190 [ 6648.226619] _raw_spin_lock+0x21/0x30 [ 6648.228095] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6648.232894] obd_mod_free+0x46/0x140 [ptlrpc] [ 6648.236090] mdc_commit_open+0x91/0xb0 [mdc] [ 6648.240195] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6648.245551] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6648.250821] after_reply+0xe2c/0x1780 [ptlrpc] [ 6648.261917] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6648.267020] ? __next_timer_interrupt+0x160/0x160 [ 6648.270032] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6648.273042] ? do_wait_intr+0xf0/0xf0 [ 6648.275393] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6648.279939] kthread+0x1d1/0x200 [ 6648.284734] ? set_kthread_struct+0x70/0x70 [ 6648.287427] ret_from_fork+0x1f/0x30 [ 6652.060322] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 6652.062011] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 6652.062774] Modules linked in: [ 6652.066022] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6652.066187] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6652.066194] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6652.066198] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6652.066217] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6652.066221] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6652.066226] RAX: 0000000000003c81 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6652.066228] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6652.066230] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6652.066232] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6652.066234] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6652.066238] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6652.066241] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6652.066243] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6652.066253] Call Trace: [ 6652.066258] [ 6652.066264] ? show_regs.cold.9+0x22/0x2f [ 6652.066273] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6652.066285] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6652.066289] ? watchdog+0x40/0x40 [ 6652.066294] ? hrtimer_interrupt+0x128/0x2c0 [ 6652.066298] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6652.066307] ? apic_timer_interrupt+0xf/0x20 [ 6652.066317] [ 6652.066320] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6652.066325] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6652.066330] do_raw_spin_lock+0x177/0x190 [ 6652.066335] _raw_spin_lock+0x21/0x30 [ 6652.066341] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6652.066715] ? do_raw_spin_unlock+0x75/0x190 [ 6652.066722] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6652.066956] ? do_wait_intr+0xf0/0xf0 [ 6652.066964] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6652.067229] mdc_reint+0x2d/0x110 [mdc] [ 6652.067282] mdc_unlink+0x658/0x870 [mdc] [ 6652.067319] lmv_unlink+0x34d/0xbd0 [lmv] [ 6652.067353] ll_unlink+0x24d/0x7f0 [lustre] [ 6652.071570] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) [ 6652.089575] vfs_unlink+0x202/0x330 [ 6652.089597] do_unlinkat+0x2fa/0x460 [ 6652.089603] __x64_sys_unlink+0x37/0x50 [ 6652.089608] do_syscall_64+0xc1/0x440 [ 6652.089615] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6652.089625] RIP: 0033:0x7f83e5f5c10b [ 6652.089633] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6652.089637] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6652.089643] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6652.089646] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6652.089648] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6652.089651] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6652.089653] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6652.423933] fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6652.458706] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6652.472456] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6652.480277] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6652.485307] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6652.503843] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6652.514140] RAX: 0000000000007583 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6652.525013] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6652.526732] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6652.537187] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6652.553726] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6652.565245] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6652.567334] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6652.591378] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6652.597881] Call Trace: [ 6652.604479] [ 6652.605082] ? show_regs.cold.9+0x22/0x2f [ 6652.606124] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6652.612489] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6652.617288] ? watchdog+0x40/0x40 [ 6652.621402] ? hrtimer_interrupt+0x128/0x2c0 [ 6652.622534] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6652.628202] ? apic_timer_interrupt+0xf/0x20 [ 6652.632488] [ 6652.635741] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6652.642781] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6652.646785] do_raw_spin_lock+0x177/0x190 [ 6652.650850] _raw_spin_lock+0x21/0x30 [ 6652.654756] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6652.661221] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6652.662818] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6652.672548] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6652.677479] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6652.684941] ? woken_wake_function+0x30/0x30 [ 6652.689503] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6652.697785] kthread+0x1d1/0x200 [ 6652.701494] ? set_kthread_struct+0x70/0x70 [ 6652.703465] ret_from_fork+0x1f/0x30 [ 6652.707547] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4139s! [ 6652.711096] Showing busy workqueues and worker pools: [ 6652.713421] workqueue events: flags=0x0 [ 6652.715232] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6652.718932] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6652.723114] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6652.725976] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6652.729214] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6652.732378] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6652.736139] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6652.739147] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6652.743892] workqueue events_long: flags=0x0 [ 6652.746368] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6652.749787] pending: lu_objects_flush [obdclass] [ 6652.751668] workqueue events_unbound: flags=0x2 [ 6652.753257] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6652.756068] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6652.758836] workqueue events_power_efficient: flags=0x80 [ 6652.761509] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6652.764870] pending: neigh_periodic_work, do_cache_clean [ 6652.767557] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6652.771608] pending: neigh_periodic_work, check_lifetime [ 6652.774248] workqueue mm_percpu_wq: flags=0x8 [ 6652.776383] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6652.779801] pending: vmstat_update, lru_add_drain_per_cpu [ 6652.782539] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6652.785594] pending: vmstat_update [ 6652.787279] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6652.790360] pending: lru_add_drain_per_cpu BAR(48) [ 6652.792884] workqueue ipv6_addrconf: flags=0x40008 [ 6652.794887] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6652.797882] pending: addrconf_verify_work [ 6652.799840] workqueue knbd0-recv: flags=0x1a [ 6652.801791] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6652.805159] in-flight: 650:recv_work [ 6652.806966] workqueue ptlrpc_pinger: flags=0x2 [ 6652.809026] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6652.811862] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6652.815447] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4139s workers=4 idle: 36 466 29757 [ 6652.819278] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6642s workers=2 idle: 652 [ 6672.061900] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6672.066336] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6672.095370] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6672.100721] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6672.104854] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6672.108629] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6672.111302] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6672.119221] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6672.124078] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6672.128512] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6672.131929] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6672.135972] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6672.140108] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6672.143558] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6672.147628] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6672.150346] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6672.154302] Call Trace: [ 6672.155985] [ 6672.157302] ? show_regs.cold.9+0x22/0x2f [ 6672.159932] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6672.161874] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6672.164508] ? watchdog+0x40/0x40 [ 6672.165870] ? hrtimer_interrupt+0x128/0x2c0 [ 6672.167828] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6672.170099] ? apic_timer_interrupt+0xf/0x20 [ 6672.171985] [ 6672.173403] ? native_safe_halt+0x2e/0x50 [ 6672.175420] kvm_wait+0xcd/0xe0 [ 6672.176968] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6672.179542] do_raw_spin_lock+0x177/0x190 [ 6672.181530] _raw_spin_lock+0x21/0x30 [ 6672.183292] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6672.186206] process_one_work+0x2c8/0x700 [ 6672.187869] worker_thread+0x296/0x6e0 [ 6672.189160] ? rescuer_thread+0x570/0x570 [ 6672.191049] kthread+0x1d1/0x200 [ 6672.193391] ? set_kthread_struct+0x70/0x70 [ 6672.195675] ret_from_fork+0x1f/0x30 [ 6680.057283] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 6680.059029] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 6680.063032] Modules linked in: lustre(O) [ 6680.065197] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 6680.065210] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6680.065280] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6680.065286] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6680.065290] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6680.065305] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6680.065309] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6680.065314] RAX: 0000000000005ed5 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6680.065316] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6680.065318] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6680.065320] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6680.065322] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6680.065325] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6680.065328] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6680.065330] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6680.065339] Call Trace: [ 6680.065344] [ 6680.065350] ? show_regs.cold.9+0x22/0x2f [ 6680.065360] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6680.065367] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6680.065371] ? watchdog+0x40/0x40 [ 6680.065376] ? hrtimer_interrupt+0x128/0x2c0 [ 6680.065380] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6680.065389] ? apic_timer_interrupt+0xf/0x20 [ 6680.065393] [ 6680.065396] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6680.065401] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6680.065405] do_raw_spin_lock+0x177/0x190 [ 6680.065410] _raw_spin_lock+0x21/0x30 [ 6680.065419] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6680.065687] ? do_raw_spin_unlock+0x75/0x190 [ 6680.065693] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6680.065889] ? do_wait_intr+0xf0/0xf0 [ 6680.065897] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6680.067087] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6680.067173] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6680.067180] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6680.067182] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6680.067192] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6680.067196] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6680.067201] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6680.067203] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6680.067205] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6680.067208] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6680.067210] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6680.067214] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6680.067217] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6680.067219] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6680.067227] Call Trace: [ 6680.067231] [ 6680.067235] ? show_regs.cold.9+0x22/0x2f [ 6680.067244] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6680.067250] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6680.067254] ? watchdog+0x40/0x40 [ 6680.067259] ? hrtimer_interrupt+0x128/0x2c0 [ 6680.067263] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6680.067270] ? apic_timer_interrupt+0xf/0x20 [ 6680.067274] [ 6680.067277] ? native_safe_halt+0x2e/0x50 [ 6680.067281] kvm_wait+0xcd/0xe0 [ 6680.067289] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6680.067297] do_raw_spin_lock+0x177/0x190 [ 6680.067301] _raw_spin_lock+0x21/0x30 [ 6680.067306] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6680.067807] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6680.068042] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6680.068302] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6680.068732] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6680.068966] ? woken_wake_function+0x30/0x30 [ 6680.068975] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6680.070002] kthread+0x1d1/0x200 [ 6680.070011] ? set_kthread_struct+0x70/0x70 [ 6680.070014] ret_from_fork+0x1f/0x30 [ 6680.070567] mdc_reint+0x2d/0x110 [mdc] [ 6680.070607] mdc_unlink+0x658/0x870 [mdc] [ 6680.070635] lmv_unlink+0x34d/0xbd0 [lmv] [ 6680.070676] ll_unlink+0x24d/0x7f0 [lustre] [ 6680.072803] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 [ 6680.087353] vfs_unlink+0x202/0x330 [ 6680.087375] do_unlinkat+0x2fa/0x460 [ 6680.087381] __x64_sys_unlink+0x37/0x50 [ 6680.087385] do_syscall_64+0xc1/0x440 [ 6680.087392] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6680.087400] RIP: 0033:0x7f83e5f5c10b [ 6680.087409] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6680.087414] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6680.087419] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6680.087422] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6680.087424] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6680.087426] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6680.087429] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6680.636524] dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6680.653819] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6680.659945] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6680.664196] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6680.667775] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6680.681968] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6680.686996] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6680.690164] RDX: 0000000000007651 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6680.693588] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6680.696832] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6680.700496] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6680.703533] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6680.706689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6680.708703] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6680.710999] Call Trace: [ 6680.711817] [ 6680.712699] ? show_regs.cold.9+0x22/0x2f [ 6680.714117] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6680.715889] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6680.717273] ? watchdog+0x40/0x40 [ 6680.718650] ? hrtimer_interrupt+0x128/0x2c0 [ 6680.720316] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6680.721739] ? apic_timer_interrupt+0xf/0x20 [ 6680.723501] [ 6680.723988] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6680.727024] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6680.734123] do_raw_spin_lock+0x177/0x190 [ 6680.736747] _raw_spin_lock+0x21/0x30 [ 6680.738068] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6680.744738] obd_mod_free+0x46/0x140 [ptlrpc] [ 6680.751898] mdc_commit_open+0x91/0xb0 [mdc] [ 6680.754752] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6680.758770] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6680.763885] after_reply+0xe2c/0x1780 [ptlrpc] [ 6680.767250] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6680.777201] ? __next_timer_interrupt+0x160/0x160 [ 6680.782942] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6680.785168] ? do_wait_intr+0xf0/0xf0 [ 6680.788304] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6680.794823] kthread+0x1d1/0x200 [ 6680.801098] ? set_kthread_struct+0x70/0x70 [ 6680.804929] ret_from_fork+0x1f/0x30 [ 6683.109152] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4169s! [ 6683.112667] Showing busy workqueues and worker pools: [ 6683.115110] workqueue events: flags=0x0 [ 6683.116547] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6683.119539] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6683.123796] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6683.127151] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6683.129820] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6683.132288] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6683.135517] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6683.138424] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6683.142150] workqueue events_long: flags=0x0 [ 6683.143949] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6683.146293] pending: lu_objects_flush [obdclass] [ 6683.148620] workqueue events_unbound: flags=0x2 [ 6683.150577] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6683.153417] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6683.155781] workqueue events_power_efficient: flags=0x80 [ 6683.158520] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6683.161168] pending: neigh_periodic_work, do_cache_clean [ 6683.163291] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6683.165961] pending: neigh_periodic_work, check_lifetime [ 6683.168202] workqueue mm_percpu_wq: flags=0x8 [ 6683.170247] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6683.173032] pending: vmstat_update, lru_add_drain_per_cpu [ 6683.175810] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6683.178130] pending: vmstat_update [ 6683.179666] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6683.182036] pending: lru_add_drain_per_cpu BAR(48) [ 6683.184919] workqueue ipv6_addrconf: flags=0x40008 [ 6683.186867] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6683.190260] pending: addrconf_verify_work [ 6683.191880] workqueue knbd0-recv: flags=0x1a [ 6683.193586] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6683.197174] in-flight: 650:recv_work [ 6683.199306] workqueue ptlrpc_pinger: flags=0x2 [ 6683.201117] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6683.204386] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6683.206842] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4170s workers=4 idle: 36 466 29757 [ 6683.210597] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6673s workers=2 idle: 652 [ 6700.061377] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6700.068167] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6700.114556] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6700.129508] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6700.139628] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6700.144544] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6700.150603] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6700.167108] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6700.173470] RAX: 00000000000003d1 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6700.178316] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6700.183940] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6700.193825] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6700.203821] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6700.210697] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6700.221907] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6700.227236] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6700.234529] Call Trace: [ 6700.237371] [ 6700.240342] ? show_regs.cold.9+0x22/0x2f [ 6700.244565] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6700.249007] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6700.251785] ? watchdog+0x40/0x40 [ 6700.254333] ? hrtimer_interrupt+0x128/0x2c0 [ 6700.256259] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6700.260793] ? apic_timer_interrupt+0xf/0x20 [ 6700.266465] [ 6700.269366] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6700.276251] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6700.278655] do_raw_spin_lock+0x177/0x190 [ 6700.281580] _raw_spin_lock+0x21/0x30 [ 6700.283628] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6700.285908] process_one_work+0x2c8/0x700 [ 6700.289485] worker_thread+0x296/0x6e0 [ 6700.292589] ? rescuer_thread+0x570/0x570 [ 6700.296647] kthread+0x1d1/0x200 [ 6700.301274] ? set_kthread_struct+0x70/0x70 [ 6700.305396] ret_from_fork+0x1f/0x30 [ 6708.058252] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 6708.059027] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 6708.062021] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6708.062037] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6708.062136] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6708.062143] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6708.062146] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6708.062162] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6708.062169] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6708.062177] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6708.062180] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6708.062182] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6708.062184] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6708.062190] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6708.062195] Modules linked in: [ 6708.062195] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6708.062199] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6708.062265] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6708.062270] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6708.062273] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6708.062280] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6708.062284] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6708.062287] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6708.062290] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6708.062292] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6708.062295] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6708.062297] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6708.062300] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6708.062303] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6708.062305] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6708.062313] Call Trace: [ 6708.062317] [ 6708.062322] ? show_regs.cold.9+0x22/0x2f [ 6708.062331] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6708.062338] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6708.062342] ? watchdog+0x40/0x40 [ 6708.062346] ? hrtimer_interrupt+0x128/0x2c0 [ 6708.062350] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6708.062356] ? apic_timer_interrupt+0xf/0x20 [ 6708.062360] [ 6708.062363] ? native_safe_halt+0x2e/0x50 [ 6708.062367] kvm_wait+0xcd/0xe0 [ 6708.062374] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6708.062380] do_raw_spin_lock+0x177/0x190 [ 6708.062384] _raw_spin_lock+0x21/0x30 [ 6708.062389] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6708.062595] obd_mod_free+0x46/0x140 [ptlrpc] [ 6708.062719] mdc_commit_open+0x91/0xb0 [mdc] [ 6708.062746] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6708.062935] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6708.063130] after_reply+0xe2c/0x1780 [ptlrpc] [ 6708.063321] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6708.063504] ? __next_timer_interrupt+0x160/0x160 [ 6708.063511] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6708.063644] ? do_wait_intr+0xf0/0xf0 [ 6708.063650] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6708.063791] kthread+0x1d1/0x200 [ 6708.063796] ? set_kthread_struct+0x70/0x70 [ 6708.063800] ret_from_fork+0x1f/0x30 [ 6708.078903] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [ 6708.083192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6708.083203] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6708.083213] Call Trace: [ 6708.083224] [ 6708.083233] ? show_regs.cold.9+0x22/0x2f [ 6708.083247] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6708.083254] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6708.083258] ? watchdog+0x40/0x40 [ 6708.083262] ? hrtimer_interrupt+0x128/0x2c0 [ 6708.083267] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6708.083273] ? apic_timer_interrupt+0xf/0x20 [ 6708.083278] [ 6708.083281] ? native_safe_halt+0x2e/0x50 [ 6708.083284] kvm_wait+0xcd/0xe0 [ 6708.083292] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6708.083300] do_raw_spin_lock+0x177/0x190 [ 6708.083305] _raw_spin_lock+0x21/0x30 [ 6708.083311] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6708.083791] ? do_raw_spin_unlock+0x75/0x190 [ 6708.083796] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6708.084010] ? do_wait_intr+0xf0/0xf0 [ 6708.084019] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6708.084240] mdc_reint+0x2d/0x110 [mdc] [ 6708.084279] mdc_unlink+0x658/0x870 [mdc] [ 6708.084305] lmv_unlink+0x34d/0xbd0 [lmv] [ 6708.084345] ll_unlink+0x24d/0x7f0 [lustre] [ 6708.084464] vfs_unlink+0x202/0x330 [ 6708.084472] do_unlinkat+0x2fa/0x460 [ 6708.084477] __x64_sys_unlink+0x37/0x50 [ 6708.084480] do_syscall_64+0xc1/0x440 [ 6708.084493] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6708.084499] RIP: 0033:0x7f83e5f5c10b [ 6708.084506] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6708.084510] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6708.084515] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6708.084517] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6708.084519] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6708.084521] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6708.084523] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6708.650246] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6708.672511] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6708.685074] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6708.688410] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6708.690032] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6708.707682] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6708.714453] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6708.720974] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6708.724596] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6708.727551] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6708.733551] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6708.737612] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6708.741484] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6708.745450] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6708.749515] Call Trace: [ 6708.750645] [ 6708.754347] ? show_regs.cold.9+0x22/0x2f [ 6708.758405] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6708.760070] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6708.766566] ? watchdog+0x40/0x40 [ 6708.767747] ? hrtimer_interrupt+0x128/0x2c0 [ 6708.776952] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6708.785617] ? apic_timer_interrupt+0xf/0x20 [ 6708.790208] [ 6708.790961] ? native_safe_halt+0x2e/0x50 [ 6708.799963] kvm_wait+0xcd/0xe0 [ 6708.803401] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6708.806933] do_raw_spin_lock+0x177/0x190 [ 6708.808456] _raw_spin_lock+0x21/0x30 [ 6708.811657] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6708.815852] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6708.817415] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6708.820611] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6708.823548] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6708.827403] ? woken_wake_function+0x30/0x30 [ 6708.833461] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6708.838048] kthread+0x1d1/0x200 [ 6708.839897] ? set_kthread_struct+0x70/0x70 [ 6708.841587] ret_from_fork+0x1f/0x30 [ 6713.312087] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4200s! [ 6713.314941] Showing busy workqueues and worker pools: [ 6713.317309] workqueue events: flags=0x0 [ 6713.318796] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6713.321670] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6713.325524] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6713.327723] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6713.330394] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6713.332961] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6713.336458] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6713.339270] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6713.343103] workqueue events_long: flags=0x0 [ 6713.344664] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6713.347343] pending: lu_objects_flush [obdclass] [ 6713.349684] workqueue events_unbound: flags=0x2 [ 6713.351808] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6713.356240] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6713.359902] workqueue events_power_efficient: flags=0x80 [ 6713.362698] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6713.366843] pending: neigh_periodic_work, do_cache_clean [ 6713.369602] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6713.372450] pending: neigh_periodic_work, check_lifetime [ 6713.375266] workqueue mm_percpu_wq: flags=0x8 [ 6713.379604] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6713.383985] pending: vmstat_update, lru_add_drain_per_cpu [ 6713.386741] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6713.389559] pending: vmstat_update [ 6713.390738] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6713.394818] pending: lru_add_drain_per_cpu BAR(48) [ 6713.397217] workqueue ipv6_addrconf: flags=0x40008 [ 6713.399611] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6713.402154] pending: addrconf_verify_work [ 6713.403594] workqueue knbd0-recv: flags=0x1a [ 6713.405530] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6713.408240] in-flight: 650:recv_work [ 6713.410192] workqueue ptlrpc_pinger: flags=0x2 [ 6713.412189] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6713.414594] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6713.417138] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4200s workers=4 idle: 36 466 29757 [ 6713.420414] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6703s workers=2 idle: 652 [ 6721.068324] rcu: INFO: rcu_sched self-detected stall on CPU [ 6721.071474] rcu: 0-....: (3011171 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 6721.077647] (t=4740118 jiffies g=262441 q=58533) [ 6721.080421] rcu: rcu_sched kthread starved for 4209305 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 6721.085503] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6721.090376] rcu: RCU grace-period kthread stack dump: [ 6721.092370] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6721.096578] Call Trace: [ 6721.097818] __schedule+0x351/0xcb0 [ 6721.099599] schedule+0xc0/0x180 [ 6721.101174] schedule_timeout+0xb4/0x190 [ 6721.103547] ? do_raw_spin_unlock+0x75/0x190 [ 6721.106060] ? __next_timer_interrupt+0x160/0x160 [ 6721.108407] ? prepare_to_swait_event+0xf3/0x140 [ 6721.110610] rcu_gp_kthread+0x8dd/0x15b0 [ 6721.112667] ? force_qs_rnp+0x2c0/0x2c0 [ 6721.114597] kthread+0x1d1/0x200 [ 6721.116306] ? set_kthread_struct+0x70/0x70 [ 6721.118257] ret_from_fork+0x1f/0x30 [ 6721.120055] rcu: Stack dump where RCU GP kthread last ran: [ 6721.122691] Sending NMI from CPU 0 to CPUs 2: [ 6721.126358] NMI backtrace for cpu 2 [ 6721.126366] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6721.126369] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6721.126370] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6721.126373] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6721.126375] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6721.126377] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 [ 6721.126379] RAX: 000000000000727c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6721.126381] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6721.126382] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6721.126384] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6721.126385] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6721.126387] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6721.126388] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6721.126390] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6721.126391] Call Trace: [ 6721.126392] [ 6721.126393] ? show_regs.cold.9+0x22/0x2f [ 6721.126394] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6721.126395] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6721.126396] ? nmi_handle+0xbd/0x210 [ 6721.126397] ? default_do_nmi+0x4b/0x1b0 [ 6721.126399] ? do_nmi+0x280/0x380 [ 6721.126399] ? end_repeat_nmi+0x16/0x1f [ 6721.126401] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6721.126402] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6721.126403] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6721.126404] [ 6721.126406] do_raw_spin_lock+0x177/0x190 [ 6721.126407] _raw_spin_lock+0x21/0x30 [ 6721.126408] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6721.126410] process_one_work+0x2c8/0x700 [ 6721.126411] worker_thread+0x296/0x6e0 [ 6721.126412] ? rescuer_thread+0x570/0x570 [ 6721.126413] kthread+0x1d1/0x200 [ 6721.126415] ? set_kthread_struct+0x70/0x70 [ 6721.126416] ret_from_fork+0x1f/0x30 [ 6721.127036] NMI backtrace for cpu 0 [ 6721.265723] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6721.278264] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6721.284253] Call Trace: [ 6721.285458] [ 6721.288298] ? dump_stack+0xbb/0x10e [ 6721.290225] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6721.295223] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6721.297818] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6721.301366] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6721.304489] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6721.307078] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6721.309205] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6721.312388] ? update_process_times+0x6d/0xc0 [ 6721.314408] ? tick_sched_handle.isra.18+0x27/0x90 [ 6721.320352] ? tick_sched_timer+0x53/0xb0 [ 6721.321466] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6721.324078] ? tick_sched_handle.isra.18+0x90/0x90 [ 6721.327210] ? hrtimer_interrupt+0x128/0x2c0 [ 6721.329699] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6721.331386] ? apic_timer_interrupt+0xf/0x20 [ 6721.333659] [ 6721.334746] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6721.337696] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6721.340518] ? do_raw_spin_lock+0x177/0x190 [ 6721.342648] ? _raw_spin_lock+0x21/0x30 [ 6721.344549] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6721.349039] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6721.351823] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6721.354209] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6721.359296] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6721.362089] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6721.366506] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6721.370279] ? __next_timer_interrupt+0x160/0x160 [ 6721.372343] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6721.376320] ? do_wait_intr+0xf0/0xf0 [ 6721.377963] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6721.384380] ? kthread+0x1d1/0x200 [ 6721.386110] ? set_kthread_struct+0x70/0x70 [ 6721.387880] ? ret_from_fork+0x1f/0x30 [ 6736.059031] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 6736.062135] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6736.063915] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6736.083213] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6736.083230] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6736.083233] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6736.083253] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6736.083257] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6736.083262] RAX: 0000000000002a04 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6736.083265] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6736.083267] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6736.083269] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6736.083271] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6736.083275] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6736.083277] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6736.083280] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6736.083289] Call Trace: [ 6736.083296] [ 6736.083303] ? show_regs.cold.9+0x22/0x2f [ 6736.083313] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6736.083320] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6736.083324] ? watchdog+0x40/0x40 [ 6736.083329] ? hrtimer_interrupt+0x128/0x2c0 [ 6736.083334] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6736.083342] ? apic_timer_interrupt+0xf/0x20 [ 6736.083347] [ 6736.083350] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6736.083356] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6736.083365] do_raw_spin_lock+0x177/0x190 [ 6736.083370] _raw_spin_lock+0x21/0x30 [ 6736.083375] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6736.088460] Modules linked in: [ 6736.094901] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6736.095196] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6736.095420] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6736.095900] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6736.096120] ? woken_wake_function+0x30/0x30 [ 6736.096132] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6736.096355] kthread+0x1d1/0x200 [ 6736.096361] ? set_kthread_struct+0x70/0x70 [ 6736.096365] ret_from_fork+0x1f/0x30 [ 6736.297279] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6736.352770] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6736.368389] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6736.384007] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6736.386086] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6736.419523] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6736.426697] RAX: 0000000000001708 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6736.431530] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6736.436383] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6736.443488] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6736.454147] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6736.463770] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6736.477456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6736.483700] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6736.492637] Call Trace: [ 6736.494166] [ 6736.496008] ? show_regs.cold.9+0x22/0x2f [ 6736.498519] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6736.503393] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6736.506878] ? watchdog+0x40/0x40 [ 6736.509100] ? hrtimer_interrupt+0x128/0x2c0 [ 6736.512262] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6736.514411] ? apic_timer_interrupt+0xf/0x20 [ 6736.518073] [ 6736.520249] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6736.527162] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6736.532445] do_raw_spin_lock+0x177/0x190 [ 6736.537796] _raw_spin_lock+0x21/0x30 [ 6736.539128] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6736.541097] ? do_raw_spin_unlock+0x75/0x190 [ 6736.544252] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6736.546318] ? do_wait_intr+0xf0/0xf0 [ 6736.556533] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6736.564381] mdc_reint+0x2d/0x110 [mdc] [ 6736.575066] mdc_unlink+0x658/0x870 [mdc] [ 6736.577507] lmv_unlink+0x34d/0xbd0 [lmv] [ 6736.583324] ll_unlink+0x24d/0x7f0 [lustre] [ 6736.585464] vfs_unlink+0x202/0x330 [ 6736.590245] do_unlinkat+0x2fa/0x460 [ 6736.594556] __x64_sys_unlink+0x37/0x50 [ 6736.599135] do_syscall_64+0xc1/0x440 [ 6736.602581] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6736.609098] RIP: 0033:0x7f83e5f5c10b [ 6736.617651] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6736.646349] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6736.653936] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6736.660126] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6736.665976] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6736.677380] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6736.690732] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6743.522351] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4230s! [ 6743.527566] Showing busy workqueues and worker pools: [ 6743.533289] workqueue events: flags=0x0 [ 6743.534839] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6743.539348] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6743.545374] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6743.548348] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6743.552163] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6743.554863] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6743.561293] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6743.565229] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6743.569402] workqueue events_long: flags=0x0 [ 6743.571368] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6743.574280] pending: lu_objects_flush [obdclass] [ 6743.577014] workqueue events_unbound: flags=0x2 [ 6743.579013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6743.582132] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6743.585184] workqueue events_power_efficient: flags=0x80 [ 6743.587594] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6743.591318] pending: neigh_periodic_work, do_cache_clean [ 6743.594177] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6743.596426] pending: neigh_periodic_work, check_lifetime [ 6743.597917] workqueue mm_percpu_wq: flags=0x8 [ 6743.599402] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6743.601650] pending: vmstat_update, lru_add_drain_per_cpu [ 6743.603336] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6743.605097] pending: vmstat_update [ 6743.606060] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6743.607952] pending: lru_add_drain_per_cpu BAR(48) [ 6743.609302] workqueue ipv6_addrconf: flags=0x40008 [ 6743.610446] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6743.612015] pending: addrconf_verify_work [ 6743.613243] workqueue knbd0-recv: flags=0x1a [ 6743.614486] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6743.616357] in-flight: 650:recv_work [ 6743.617419] workqueue ptlrpc_pinger: flags=0x2 [ 6743.618666] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6743.620100] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6743.621642] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4230s workers=4 idle: 36 466 29757 [ 6743.623477] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6733s workers=2 idle: 652 [ 6748.064721] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6748.072608] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6748.110382] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6748.122426] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6748.128579] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6748.133009] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6748.136009] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6748.149347] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6748.157604] RAX: 0000000000004988 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6748.166214] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6748.173227] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6748.179114] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6748.183086] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6748.187645] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6748.192474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6748.195595] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6748.199920] Call Trace: [ 6748.201923] [ 6748.203154] ? show_regs.cold.9+0x22/0x2f [ 6748.205390] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6748.207296] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6748.209035] ? watchdog+0x40/0x40 [ 6748.211345] ? hrtimer_interrupt+0x128/0x2c0 [ 6748.213198] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6748.216184] ? apic_timer_interrupt+0xf/0x20 [ 6748.218565] [ 6748.219687] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6748.222017] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6748.223870] do_raw_spin_lock+0x177/0x190 [ 6748.226112] _raw_spin_lock+0x21/0x30 [ 6748.227362] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6748.229529] process_one_work+0x2c8/0x700 [ 6748.231007] worker_thread+0x296/0x6e0 [ 6748.232890] ? rescuer_thread+0x570/0x570 [ 6748.234883] kthread+0x1d1/0x200 [ 6748.236671] ? set_kthread_struct+0x70/0x70 [ 6748.240942] ret_from_fork+0x1f/0x30 [ 6764.060475] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 6764.062009] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6764.069013] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) [ 6764.073491] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6764.073587] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6764.073595] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6764.073598] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30a/0x6d0 [ 6764.073612] Code: 03 00 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 [ 6764.073620] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6764.073625] RAX: 0000000000000421 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6764.073628] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6764.073630] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6764.073632] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6764.073637] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6764.073641] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6764.073644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6764.073646] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6764.073654] Call Trace: [ 6764.073663] [ 6764.073669] ? show_regs.cold.9+0x22/0x2f [ 6764.073681] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6764.073687] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6764.073692] ? watchdog+0x40/0x40 [ 6764.073700] ? hrtimer_interrupt+0x128/0x2c0 [ 6764.073705] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6764.073721] ? apic_timer_interrupt+0xf/0x20 [ 6764.073729] [ 6764.073738] ? __pv_queued_spin_lock_slowpath+0x30a/0x6d0 [ 6764.073746] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6764.073755] do_raw_spin_lock+0x177/0x190 [ 6764.073760] _raw_spin_lock+0x21/0x30 [ 6764.073771] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6764.074230] ? do_raw_spin_unlock+0x75/0x190 [ 6764.074241] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6764.074514] ? do_wait_intr+0xf0/0xf0 [ 6764.074523] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6764.074767] mdc_reint+0x2d/0x110 [mdc] [ 6764.074820] mdc_unlink+0x658/0x870 [mdc] [ 6764.074849] lmv_unlink+0x34d/0xbd0 [lmv] [ 6764.074890] ll_unlink+0x24d/0x7f0 [lustre] [ 6764.075057] vfs_unlink+0x202/0x330 [ 6764.075072] do_unlinkat+0x2fa/0x460 [ 6764.075078] __x64_sys_unlink+0x37/0x50 [ 6764.075081] do_syscall_64+0xc1/0x440 [ 6764.075091] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6764.075098] RIP: 0033:0x7f83e5f5c10b [ 6764.075112] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6764.075116] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6764.075121] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6764.075124] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6764.075126] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6764.075128] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6764.075130] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6764.430669] fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6764.449168] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6764.456030] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6764.459312] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6764.461063] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6764.474689] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6764.477987] RAX: 0000000000000710 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6764.480379] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6764.484093] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6764.486101] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6764.489101] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6764.492197] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6764.495921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6764.498159] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6764.501822] Call Trace: [ 6764.503151] [ 6764.504101] ? show_regs.cold.9+0x22/0x2f [ 6764.505677] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6764.508160] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6764.512982] ? watchdog+0x40/0x40 [ 6764.516791] ? hrtimer_interrupt+0x128/0x2c0 [ 6764.519926] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6764.522992] ? apic_timer_interrupt+0xf/0x20 [ 6764.525457] [ 6764.526630] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6764.529647] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6764.532890] do_raw_spin_lock+0x177/0x190 [ 6764.535277] _raw_spin_lock+0x21/0x30 [ 6764.537951] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6764.541857] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6764.544635] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6764.548517] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6764.551933] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6764.555220] ? woken_wake_function+0x30/0x30 [ 6764.557845] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6764.561271] kthread+0x1d1/0x200 [ 6764.562493] ? set_kthread_struct+0x70/0x70 [ 6764.563952] ret_from_fork+0x1f/0x30 [ 6768.057919] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 6768.073728] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6768.117003] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6768.126386] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6768.141976] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6768.146184] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6768.171331] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6768.181757] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6768.188146] RDX: 0000000000007e7f RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6768.197860] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6768.217174] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6768.225549] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6768.234685] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6768.247880] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6768.252458] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6768.265896] Call Trace: [ 6768.268681] [ 6768.271012] ? show_regs.cold.9+0x22/0x2f [ 6768.279149] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6768.284932] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6768.290078] ? watchdog+0x40/0x40 [ 6768.295152] ? hrtimer_interrupt+0x128/0x2c0 [ 6768.302088] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6768.308982] ? apic_timer_interrupt+0xf/0x20 [ 6768.315781] [ 6768.316621] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6768.323692] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6768.329445] do_raw_spin_lock+0x177/0x190 [ 6768.333427] _raw_spin_lock+0x21/0x30 [ 6768.335274] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6768.338706] obd_mod_free+0x46/0x140 [ptlrpc] [ 6768.344187] mdc_commit_open+0x91/0xb0 [mdc] [ 6768.350032] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6768.358109] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6768.363775] after_reply+0xe2c/0x1780 [ptlrpc] [ 6768.365463] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6768.375113] ? __next_timer_interrupt+0x160/0x160 [ 6768.380917] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6768.386888] ? do_wait_intr+0xf0/0xf0 [ 6768.391221] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6768.398928] kthread+0x1d1/0x200 [ 6768.406123] ? set_kthread_struct+0x70/0x70 [ 6768.415098] ret_from_fork+0x1f/0x30 [ 6773.728529] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4260s! [ 6773.745862] Showing busy workqueues and worker pools: [ 6773.760014] workqueue events: flags=0x0 [ 6773.762978] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6773.771947] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6773.783310] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6773.789142] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6773.794179] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6773.800096] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6773.806963] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6773.815390] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6773.819223] workqueue events_long: flags=0x0 [ 6773.825184] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6773.832862] pending: lu_objects_flush [obdclass] [ 6773.836888] workqueue events_unbound: flags=0x2 [ 6773.842131] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6773.848058] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6773.857539] workqueue events_power_efficient: flags=0x80 [ 6773.864966] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6773.869229] pending: neigh_periodic_work, do_cache_clean [ 6773.875173] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6773.888867] pending: neigh_periodic_work, check_lifetime [ 6773.892802] workqueue mm_percpu_wq: flags=0x8 [ 6773.895869] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6773.901967] pending: vmstat_update, lru_add_drain_per_cpu [ 6773.907285] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6773.912977] pending: vmstat_update [ 6773.915221] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6773.928158] pending: lru_add_drain_per_cpu BAR(48) [ 6773.937037] workqueue ipv6_addrconf: flags=0x40008 [ 6773.944085] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6773.951219] pending: addrconf_verify_work [ 6773.954600] workqueue knbd0-recv: flags=0x1a [ 6773.957886] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6773.963057] in-flight: 650:recv_work [ 6773.966315] workqueue ptlrpc_pinger: flags=0x2 [ 6773.973271] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6773.980719] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6773.986827] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4260s workers=4 idle: 36 466 29757 [ 6773.995875] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6764s workers=2 idle: 652 [ 6776.061566] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6776.071027] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6776.117546] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6776.134889] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6776.141905] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6776.148244] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6776.153458] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6776.168518] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6776.175557] RAX: 0000000000007de0 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6776.185819] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6776.195019] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6776.208998] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6776.218088] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6776.228430] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6776.241006] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6776.247471] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6776.256806] Call Trace: [ 6776.257883] [ 6776.260746] ? show_regs.cold.9+0x22/0x2f [ 6776.263187] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6776.267011] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6776.278829] ? watchdog+0x40/0x40 [ 6776.281284] ? hrtimer_interrupt+0x128/0x2c0 [ 6776.285951] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6776.289164] ? apic_timer_interrupt+0xf/0x20 [ 6776.295592] [ 6776.296732] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6776.311742] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6776.316969] do_raw_spin_lock+0x177/0x190 [ 6776.323599] _raw_spin_lock+0x21/0x30 [ 6776.326928] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6776.332990] process_one_work+0x2c8/0x700 [ 6776.339731] worker_thread+0x296/0x6e0 [ 6776.343557] ? rescuer_thread+0x570/0x570 [ 6776.351004] kthread+0x1d1/0x200 [ 6776.363550] ? set_kthread_struct+0x70/0x70 [ 6776.368178] ret_from_fork+0x1f/0x30 [ 6792.061042] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 6792.062221] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6792.065684] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6792.065765] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6792.065772] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6792.065775] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6792.065800] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6792.065805] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6792.065809] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6792.065812] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6792.065814] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6792.065816] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6792.065819] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6792.065822] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6792.065826] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6792.065828] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6792.065837] Call Trace: [ 6792.065842] [ 6792.065848] ? show_regs.cold.9+0x22/0x2f [ 6792.065867] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6792.065873] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6792.065878] ? watchdog+0x40/0x40 [ 6792.065883] ? hrtimer_interrupt+0x128/0x2c0 [ 6792.065886] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6792.065892] ? apic_timer_interrupt+0xf/0x20 [ 6792.065896] [ 6792.065898] ? native_safe_halt+0x2e/0x50 [ 6792.065902] kvm_wait+0xcd/0xe0 [ 6792.065909] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6792.065917] do_raw_spin_lock+0x177/0x190 [ 6792.065920] _raw_spin_lock+0x21/0x30 [ 6792.065924] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6792.071369] Modules linked in: lustre(O) [ 6792.095181] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6792.098162] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [ 6792.102661] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6792.104272] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6792.104333] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6792.104339] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6792.104341] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6792.112749] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6792.120753] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6792.124542] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6792.127533] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6792.127547] RAX: 00000000000036f5 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6792.127550] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6792.127552] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6792.127554] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6792.127556] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6792.127560] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6792.127563] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6792.127566] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6792.127576] Call Trace: [ 6792.127583] [ 6792.127588] ? show_regs.cold.9+0x22/0x2f [ 6792.127601] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6792.127609] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6792.127614] ? watchdog+0x40/0x40 [ 6792.127618] ? hrtimer_interrupt+0x128/0x2c0 [ 6792.127622] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6792.127632] ? apic_timer_interrupt+0xf/0x20 [ 6792.127637] [ 6792.127640] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6792.127647] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6792.127651] do_raw_spin_lock+0x177/0x190 [ 6792.127656] _raw_spin_lock+0x21/0x30 [ 6792.127662] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6792.127956] ? do_raw_spin_unlock+0x75/0x190 [ 6792.127962] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6792.128569] ? do_wait_intr+0xf0/0xf0 [ 6792.128578] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6792.128793] mdc_reint+0x2d/0x110 [mdc] [ 6792.128842] mdc_unlink+0x658/0x870 [mdc] [ 6792.128871] lmv_unlink+0x34d/0xbd0 [lmv] [ 6792.128902] ll_unlink+0x24d/0x7f0 [lustre] [ 6792.129040] vfs_unlink+0x202/0x330 [ 6792.129424] do_unlinkat+0x2fa/0x460 [ 6792.129430] __x64_sys_unlink+0x37/0x50 [ 6792.129433] do_syscall_64+0xc1/0x440 [ 6792.129442] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6792.129450] RIP: 0033:0x7f83e5f5c10b [ 6792.129458] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6792.129462] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6792.129467] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6792.129469] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6792.129472] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6792.129474] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6792.129476] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6792.432834] ? woken_wake_function+0x30/0x30 [ 6792.434998] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6792.437806] kthread+0x1d1/0x200 [ 6792.439515] ? set_kthread_struct+0x70/0x70 [ 6792.441385] ret_from_fork+0x1f/0x30 [ 6800.056038] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 6800.059785] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6800.117993] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6800.134008] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6800.142063] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6800.152203] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6800.188056] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6800.196952] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6800.201926] RDX: 0000000000002a0a RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6800.207643] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6800.216869] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6800.227517] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6800.240818] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6800.251969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6800.258901] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6800.266421] Call Trace: [ 6800.267724] [ 6800.268746] ? show_regs.cold.9+0x22/0x2f [ 6800.270970] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6800.275741] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6800.279390] ? watchdog+0x40/0x40 [ 6800.282432] ? hrtimer_interrupt+0x128/0x2c0 [ 6800.284353] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6800.287426] ? apic_timer_interrupt+0xf/0x20 [ 6800.291171] [ 6800.292285] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6800.294720] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6800.300637] do_raw_spin_lock+0x177/0x190 [ 6800.304267] _raw_spin_lock+0x21/0x30 [ 6800.305430] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6800.311941] obd_mod_free+0x46/0x140 [ptlrpc] [ 6800.315549] mdc_commit_open+0x91/0xb0 [mdc] [ 6800.317566] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6800.324165] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6800.327636] after_reply+0xe2c/0x1780 [ptlrpc] [ 6800.333669] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6800.337239] ? __next_timer_interrupt+0x160/0x160 [ 6800.340732] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6800.342727] ? do_wait_intr+0xf0/0xf0 [ 6800.345097] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6800.350594] kthread+0x1d1/0x200 [ 6800.352337] ? set_kthread_struct+0x70/0x70 [ 6800.354347] ret_from_fork+0x1f/0x30 [ 6804.060019] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6804.062500] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6804.107157] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6804.116686] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6804.126715] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6804.132339] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6804.137891] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6804.157394] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6804.164785] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6804.175668] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6804.180428] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6804.187431] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6804.194461] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6804.200493] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6804.206575] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6804.212021] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6804.217465] Call Trace: [ 6804.218710] [ 6804.221494] ? show_regs.cold.9+0x22/0x2f [ 6804.224377] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6804.229463] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6804.234610] ? watchdog+0x40/0x40 [ 6804.239928] ? hrtimer_interrupt+0x128/0x2c0 [ 6804.242705] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6804.249743] ? apic_timer_interrupt+0xf/0x20 [ 6804.253334] [ 6804.259160] ? native_safe_halt+0x2e/0x50 [ 6804.260678] kvm_wait+0xcd/0xe0 [ 6804.265325] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6804.269625] do_raw_spin_lock+0x177/0x190 [ 6804.274042] _raw_spin_lock+0x21/0x30 [ 6804.277290] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6804.283878] process_one_work+0x2c8/0x700 [ 6804.286833] worker_thread+0x296/0x6e0 [ 6804.290816] ? rescuer_thread+0x570/0x570 [ 6804.294781] kthread+0x1d1/0x200 [ 6804.299573] ? set_kthread_struct+0x70/0x70 [ 6804.302378] ret_from_fork+0x1f/0x30 [ 6804.455468] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4291s! [ 6804.469369] Showing busy workqueues and worker pools: [ 6804.471255] workqueue events: flags=0x0 [ 6804.474492] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6804.477453] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6804.482088] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6804.485028] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6804.488194] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6804.491224] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6804.494195] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6804.497133] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6804.501083] workqueue events_long: flags=0x0 [ 6804.502786] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6804.505288] pending: lu_objects_flush [obdclass] [ 6804.507428] workqueue events_unbound: flags=0x2 [ 6804.509236] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6804.511198] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6804.512762] workqueue events_power_efficient: flags=0x80 [ 6804.514534] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6804.516962] pending: neigh_periodic_work, do_cache_clean [ 6804.519673] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6804.529660] pending: neigh_periodic_work, check_lifetime [ 6804.533014] workqueue mm_percpu_wq: flags=0x8 [ 6804.538509] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6804.542089] pending: vmstat_update, lru_add_drain_per_cpu [ 6804.545517] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6804.548923] pending: vmstat_update [ 6804.555899] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6804.563672] pending: lru_add_drain_per_cpu BAR(48) [ 6804.568964] workqueue ipv6_addrconf: flags=0x40008 [ 6804.570820] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6804.581857] pending: addrconf_verify_work [ 6804.585386] workqueue knbd0-recv: flags=0x1a [ 6804.587415] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6804.591085] in-flight: 650:recv_work [ 6804.593020] workqueue ptlrpc_pinger: flags=0x2 [ 6804.600328] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6804.608482] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6804.611418] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4291s workers=4 idle: 36 466 29757 [ 6804.619379] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6794s workers=2 idle: 652 [ 6820.059704] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 6820.064861] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6820.065115] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6820.065188] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6820.065264] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6820.065268] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6820.065281] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6820.065284] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6820.065289] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6820.065292] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6820.065294] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6820.065296] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6820.065298] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6820.065302] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6820.065304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6820.065306] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6820.065315] Call Trace: [ 6820.065320] [ 6820.065325] ? show_regs.cold.9+0x22/0x2f [ 6820.065335] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6820.065341] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6820.065345] ? watchdog+0x40/0x40 [ 6820.065349] ? hrtimer_interrupt+0x128/0x2c0 [ 6820.065353] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6820.065359] ? apic_timer_interrupt+0xf/0x20 [ 6820.065364] [ 6820.065367] ? native_safe_halt+0x2e/0x50 [ 6820.065371] kvm_wait+0xcd/0xe0 [ 6820.065378] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6820.065387] do_raw_spin_lock+0x177/0x190 [ 6820.065392] _raw_spin_lock+0x21/0x30 [ 6820.065397] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6820.065774] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6820.065974] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6820.066185] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6820.066429] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6820.066590] ? woken_wake_function+0x30/0x30 [ 6820.066673] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6820.066895] kthread+0x1d1/0x200 [ 6820.066902] ? set_kthread_struct+0x70/0x70 [ 6820.066907] ret_from_fork+0x1f/0x30 [ 6820.337300] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6820.380111] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6820.391276] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6820.398289] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6820.401854] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6820.419948] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 6820.429131] RAX: 0000000000006f07 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6820.435900] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6820.443681] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6820.448407] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6820.458758] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6820.471168] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6820.476626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6820.483160] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6820.500086] Call Trace: [ 6820.506062] [ 6820.506959] ? show_regs.cold.9+0x22/0x2f [ 6820.508476] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6820.512244] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6820.514750] ? watchdog+0x40/0x40 [ 6820.516265] ? hrtimer_interrupt+0x128/0x2c0 [ 6820.518592] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6820.520743] ? apic_timer_interrupt+0xf/0x20 [ 6820.522879] [ 6820.523982] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6820.526647] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6820.532205] do_raw_spin_lock+0x177/0x190 [ 6820.533979] _raw_spin_lock+0x21/0x30 [ 6820.536206] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6820.540291] ? do_raw_spin_unlock+0x75/0x190 [ 6820.542713] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6820.544769] ? do_wait_intr+0xf0/0xf0 [ 6820.546946] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6820.549196] mdc_reint+0x2d/0x110 [mdc] [ 6820.551474] mdc_unlink+0x658/0x870 [mdc] [ 6820.553247] lmv_unlink+0x34d/0xbd0 [lmv] [ 6820.555286] ll_unlink+0x24d/0x7f0 [lustre] [ 6820.557202] vfs_unlink+0x202/0x330 [ 6820.558574] do_unlinkat+0x2fa/0x460 [ 6820.560364] __x64_sys_unlink+0x37/0x50 [ 6820.561846] do_syscall_64+0xc1/0x440 [ 6820.563439] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6820.565402] RIP: 0033:0x7f83e5f5c10b [ 6820.566750] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6820.578804] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6820.583058] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6820.588626] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6820.595703] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6820.598675] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6820.603439] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6832.057445] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 6832.061210] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6832.062738] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6832.080526] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6832.080543] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6832.080547] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6832.080562] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6832.080566] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6832.080571] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6832.080573] RDX: 0000000000000783 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6832.080576] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6832.080578] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6832.080580] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6832.080583] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6832.080587] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6832.080589] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6832.080598] Call Trace: [ 6832.080604] [ 6832.080610] ? show_regs.cold.9+0x22/0x2f [ 6832.080619] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6832.080626] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6832.080630] ? watchdog+0x40/0x40 [ 6832.080634] ? hrtimer_interrupt+0x128/0x2c0 [ 6832.080639] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6832.080646] ? apic_timer_interrupt+0xf/0x20 [ 6832.080650] [ 6832.080653] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6832.080658] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6832.080662] do_raw_spin_lock+0x177/0x190 [ 6832.080667] _raw_spin_lock+0x21/0x30 [ 6832.080685] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6832.081114] obd_mod_free+0x46/0x140 [ptlrpc] [ 6832.081303] mdc_commit_open+0x91/0xb0 [mdc] [ 6832.081348] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6832.081535] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6832.081720] after_reply+0xe2c/0x1780 [ptlrpc] [ 6832.082028] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6832.082221] ? __next_timer_interrupt+0x160/0x160 [ 6832.082231] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6832.082431] ? do_wait_intr+0xf0/0xf0 [ 6832.082441] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6832.082637] kthread+0x1d1/0x200 [ 6832.082643] ? set_kthread_struct+0x70/0x70 [ 6832.082647] ret_from_fork+0x1f/0x30 [ 6832.346045] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6832.400458] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6832.411750] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6832.418014] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6832.424023] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6832.427485] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6832.460186] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6832.471255] RAX: 000000000000735d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6832.479208] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6832.484195] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6832.495579] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6832.509283] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6832.518444] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6832.526194] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6832.537026] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6832.542243] Call Trace: [ 6832.544342] [ 6832.546089] ? show_regs.cold.9+0x22/0x2f [ 6832.549210] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6832.551300] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6832.554445] ? watchdog+0x40/0x40 [ 6832.559816] ? hrtimer_interrupt+0x128/0x2c0 [ 6832.564371] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6832.570051] ? apic_timer_interrupt+0xf/0x20 [ 6832.576339] [ 6832.580204] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6832.588725] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6832.593300] do_raw_spin_lock+0x177/0x190 [ 6832.598161] _raw_spin_lock+0x21/0x30 [ 6832.602279] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6832.614075] process_one_work+0x2c8/0x700 [ 6832.618382] worker_thread+0x296/0x6e0 [ 6832.623666] ? rescuer_thread+0x570/0x570 [ 6832.625578] kthread+0x1d1/0x200 [ 6832.629742] ? set_kthread_struct+0x70/0x70 [ 6832.633350] ret_from_fork+0x1f/0x30 [ 6834.658943] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4321s! [ 6834.665612] Showing busy workqueues and worker pools: [ 6834.667752] workqueue events: flags=0x0 [ 6834.669466] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6834.673042] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6834.677190] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6834.679809] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6834.682927] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6834.686146] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6834.690071] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6834.693763] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6834.697709] workqueue events_long: flags=0x0 [ 6834.699786] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6834.702799] pending: lu_objects_flush [obdclass] [ 6834.704419] workqueue events_unbound: flags=0x2 [ 6834.705721] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6834.708068] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6834.710644] workqueue events_power_efficient: flags=0x80 [ 6834.713077] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6834.716075] pending: neigh_periodic_work, do_cache_clean [ 6834.718792] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6834.721917] pending: neigh_periodic_work, check_lifetime [ 6834.725111] workqueue mm_percpu_wq: flags=0x8 [ 6834.727518] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6834.730309] pending: vmstat_update, lru_add_drain_per_cpu [ 6834.733346] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6834.736477] pending: vmstat_update [ 6834.738353] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6834.742759] pending: lru_add_drain_per_cpu BAR(48) [ 6834.745606] workqueue ipv6_addrconf: flags=0x40008 [ 6834.747198] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6834.750439] pending: addrconf_verify_work [ 6834.753084] workqueue knbd0-recv: flags=0x1a [ 6834.755891] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6834.758845] in-flight: 650:recv_work [ 6834.760784] workqueue ptlrpc_pinger: flags=0x2 [ 6834.762584] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6834.765597] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6834.768686] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4321s workers=4 idle: 36 466 29757 [ 6834.773217] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6824s workers=2 idle: 652 [ 6848.059541] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 6848.065671] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6848.068067] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6848.068150] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6848.068158] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6848.068160] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6848.068182] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6848.068187] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6848.068192] RAX: 0000000000005ab9 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6848.068194] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6848.068196] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6848.068198] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6848.068201] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6848.068204] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6848.068206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6848.068209] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6848.068223] Call Trace: [ 6848.068229] [ 6848.068234] ? show_regs.cold.9+0x22/0x2f [ 6848.068244] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6848.068250] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6848.068254] ? watchdog+0x40/0x40 [ 6848.068258] ? hrtimer_interrupt+0x128/0x2c0 [ 6848.068262] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6848.068269] ? apic_timer_interrupt+0xf/0x20 [ 6848.068273] [ 6848.068276] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6848.068281] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6848.068287] do_raw_spin_lock+0x177/0x190 [ 6848.068298] _raw_spin_lock+0x21/0x30 [ 6848.068302] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6848.072651] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6848.072802] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6848.072810] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6848.072890] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6848.072904] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6848.072908] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6848.072912] RAX: 0000000000006d6a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6848.072914] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6848.072916] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6848.072918] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6848.072920] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6848.072923] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6848.072925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6848.072927] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6848.072934] Call Trace: [ 6848.072948] [ 6848.072956] ? show_regs.cold.9+0x22/0x2f [ 6848.072966] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6848.072972] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6848.102115] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6848.113654] ? watchdog+0x40/0x40 [ 6848.113674] ? hrtimer_interrupt+0x128/0x2c0 [ 6848.113680] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6848.113687] ? apic_timer_interrupt+0xf/0x20 [ 6848.113692] [ 6848.113696] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6848.113703] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6848.113707] do_raw_spin_lock+0x177/0x190 [ 6848.113714] _raw_spin_lock+0x21/0x30 [ 6848.113721] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6848.122157] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6848.122455] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6848.122985] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6848.123286] ? woken_wake_function+0x30/0x30 [ 6848.123297] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6848.124552] kthread+0x1d1/0x200 [ 6848.124566] ? set_kthread_struct+0x70/0x70 [ 6848.124571] ret_from_fork+0x1f/0x30 [ 6848.369197] ? do_raw_spin_unlock+0x75/0x190 [ 6848.370824] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6848.373559] ? do_wait_intr+0xf0/0xf0 [ 6848.379027] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6848.382311] mdc_reint+0x2d/0x110 [mdc] [ 6848.383771] mdc_unlink+0x658/0x870 [mdc] [ 6848.385870] lmv_unlink+0x34d/0xbd0 [lmv] [ 6848.387367] ll_unlink+0x24d/0x7f0 [lustre] [ 6848.389830] vfs_unlink+0x202/0x330 [ 6848.392207] do_unlinkat+0x2fa/0x460 [ 6848.396491] __x64_sys_unlink+0x37/0x50 [ 6848.404304] do_syscall_64+0xc1/0x440 [ 6848.405573] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6848.410691] RIP: 0033:0x7f83e5f5c10b [ 6848.414258] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6848.427606] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6848.435454] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6848.441334] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6848.444300] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6848.447433] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6848.450297] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6860.056088] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 6860.060020] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6860.062590] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) [ 6860.066208] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6860.066449] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6860.066458] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6860.066464] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6860.077091] fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6860.077427] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6860.077435] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6860.077438] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6860.077453] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6860.077457] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6860.077461] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6860.077464] RDX: 00000000000000b7 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6860.077467] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6860.077469] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6860.077471] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6860.077474] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6860.077477] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6860.077480] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6860.077489] Call Trace: [ 6860.077495] [ 6860.077500] ? show_regs.cold.9+0x22/0x2f [ 6860.077511] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6860.077518] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6860.077522] ? watchdog+0x40/0x40 [ 6860.077526] ? hrtimer_interrupt+0x128/0x2c0 [ 6860.077530] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6860.077538] ? apic_timer_interrupt+0xf/0x20 [ 6860.077543] [ 6860.077546] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6860.077551] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6860.077556] do_raw_spin_lock+0x177/0x190 [ 6860.077561] _raw_spin_lock+0x21/0x30 [ 6860.077567] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6860.078106] obd_mod_free+0x46/0x140 [ptlrpc] [ 6860.078317] mdc_commit_open+0x91/0xb0 [mdc] [ 6860.078355] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6860.122093] [ 6860.122103] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6860.122126] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6860.122131] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6860.122136] RAX: 0000000000002b7d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6860.122139] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6860.122141] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6860.122143] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6860.122146] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6860.122150] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6860.122152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6860.122155] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6860.122166] Call Trace: [ 6860.122172] [ 6860.122178] ? show_regs.cold.9+0x22/0x2f [ 6860.122200] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6860.122462] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6860.122467] ? watchdog+0x40/0x40 [ 6860.122471] ? hrtimer_interrupt+0x128/0x2c0 [ 6860.122475] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6860.122487] ? apic_timer_interrupt+0xf/0x20 [ 6860.122491] [ 6860.122495] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6860.122500] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6860.122508] do_raw_spin_lock+0x177/0x190 [ 6860.122513] _raw_spin_lock+0x21/0x30 [ 6860.122519] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6860.122933] process_one_work+0x2c8/0x700 [ 6860.122940] worker_thread+0x296/0x6e0 [ 6860.122945] ? rescuer_thread+0x570/0x570 [ 6860.122949] kthread+0x1d1/0x200 [ 6860.122954] ? set_kthread_struct+0x70/0x70 [ 6860.122968] ret_from_fork+0x1f/0x30 [ 6860.471086] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6860.474893] after_reply+0xe2c/0x1780 [ptlrpc] [ 6860.479201] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6860.483706] ? __next_timer_interrupt+0x160/0x160 [ 6860.485371] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6860.489248] ? do_wait_intr+0xf0/0xf0 [ 6860.494649] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6860.498125] kthread+0x1d1/0x200 [ 6860.501158] ? set_kthread_struct+0x70/0x70 [ 6860.504691] ret_from_fork+0x1f/0x30 [ 6864.866033] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4351s! [ 6864.874205] Showing busy workqueues and worker pools: [ 6864.880178] workqueue events: flags=0x0 [ 6864.883771] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6864.891865] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6864.899719] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6864.905090] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6864.907964] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6864.911612] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6864.921063] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6864.924178] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6864.930723] workqueue events_long: flags=0x0 [ 6864.932755] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6864.935051] pending: lu_objects_flush [obdclass] [ 6864.936938] workqueue events_unbound: flags=0x2 [ 6864.938322] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6864.941754] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6864.944332] workqueue events_power_efficient: flags=0x80 [ 6864.946209] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6864.951793] pending: neigh_periodic_work, do_cache_clean [ 6864.959697] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6864.967514] pending: neigh_periodic_work, check_lifetime [ 6864.970764] workqueue mm_percpu_wq: flags=0x8 [ 6864.972228] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6864.976988] pending: vmstat_update, lru_add_drain_per_cpu [ 6864.980231] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6864.985384] pending: vmstat_update [ 6864.992142] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6865.003185] pending: lru_add_drain_per_cpu BAR(48) [ 6865.008044] workqueue ipv6_addrconf: flags=0x40008 [ 6865.014398] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6865.024283] pending: addrconf_verify_work [ 6865.025991] workqueue knbd0-recv: flags=0x1a [ 6865.028509] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6865.038431] in-flight: 650:recv_work [ 6865.039930] workqueue ptlrpc_pinger: flags=0x2 [ 6865.046625] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6865.050927] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6865.053727] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4351s workers=4 idle: 36 466 29757 [ 6865.056785] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6855s workers=2 idle: 652 [ 6876.061416] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 6876.062276] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6876.070313] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6876.070404] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6876.070412] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6876.070414] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6876.070477] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6876.070485] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6876.070489] RAX: 000000000000548e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6876.070492] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6876.070494] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6876.070496] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6876.070498] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6876.070502] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6876.070506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6876.070508] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6876.070516] Call Trace: [ 6876.070533] [ 6876.070539] ? show_regs.cold.9+0x22/0x2f [ 6876.070549] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6876.070556] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6876.070561] ? watchdog+0x40/0x40 [ 6876.070566] ? hrtimer_interrupt+0x128/0x2c0 [ 6876.070570] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6876.070665] ? apic_timer_interrupt+0xf/0x20 [ 6876.070670] [ 6876.070674] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6876.070679] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6876.070685] do_raw_spin_lock+0x177/0x190 [ 6876.070690] _raw_spin_lock+0x21/0x30 [ 6876.070695] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6876.071064] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6876.071404] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6876.072668] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6876.073197] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6876.073747] ? woken_wake_function+0x30/0x30 [ 6876.073755] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6876.073902] kthread+0x1d1/0x200 [ 6876.073906] ? set_kthread_struct+0x70/0x70 [ 6876.073909] ret_from_fork+0x1f/0x30 [ 6876.306911] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6876.333918] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6876.338332] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6876.345584] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6876.351098] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6876.362838] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6876.369969] RAX: 00000000000004c0 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6876.378239] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6876.387031] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6876.388967] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6876.398475] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6876.404359] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6876.409231] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6876.413068] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6876.421204] Call Trace: [ 6876.422645] [ 6876.423606] ? show_regs.cold.9+0x22/0x2f [ 6876.427285] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6876.430737] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6876.435561] ? watchdog+0x40/0x40 [ 6876.439765] ? hrtimer_interrupt+0x128/0x2c0 [ 6876.441340] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6876.443255] ? apic_timer_interrupt+0xf/0x20 [ 6876.445059] [ 6876.445824] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6876.449205] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6876.452803] do_raw_spin_lock+0x177/0x190 [ 6876.455320] _raw_spin_lock+0x21/0x30 [ 6876.457850] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6876.461327] ? do_raw_spin_unlock+0x75/0x190 [ 6876.464099] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6876.467871] ? do_wait_intr+0xf0/0xf0 [ 6876.469900] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6876.473306] mdc_reint+0x2d/0x110 [mdc] [ 6876.475859] mdc_unlink+0x658/0x870 [mdc] [ 6876.478441] lmv_unlink+0x34d/0xbd0 [lmv] [ 6876.482960] ll_unlink+0x24d/0x7f0 [lustre] [ 6876.486268] vfs_unlink+0x202/0x330 [ 6876.496144] do_unlinkat+0x2fa/0x460 [ 6876.501089] __x64_sys_unlink+0x37/0x50 [ 6876.503591] do_syscall_64+0xc1/0x440 [ 6876.506230] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6876.508898] RIP: 0033:0x7f83e5f5c10b [ 6876.511331] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6876.529884] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6876.538438] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6876.545391] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6876.554081] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6876.557846] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6876.562861] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6888.060007] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 6888.066719] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6888.095610] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6888.100614] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6888.104481] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6888.107992] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6888.116722] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6888.132766] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6888.138959] RAX: 0000000000001071 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6888.142252] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6888.147733] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6888.154562] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6888.158061] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6888.165709] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6888.169187] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6888.173182] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6888.175087] Call Trace: [ 6888.177113] [ 6888.178002] ? show_regs.cold.9+0x22/0x2f [ 6888.180548] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6888.182679] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6888.186844] ? watchdog+0x40/0x40 [ 6888.188446] ? hrtimer_interrupt+0x128/0x2c0 [ 6888.190094] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6888.192598] ? apic_timer_interrupt+0xf/0x20 [ 6888.194473] [ 6888.198256] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6888.201930] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6888.206924] do_raw_spin_lock+0x177/0x190 [ 6888.208504] _raw_spin_lock+0x21/0x30 [ 6888.209959] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6888.212524] process_one_work+0x2c8/0x700 [ 6888.213785] worker_thread+0x296/0x6e0 [ 6888.215112] ? rescuer_thread+0x570/0x570 [ 6888.216876] kthread+0x1d1/0x200 [ 6888.218062] ? set_kthread_struct+0x70/0x70 [ 6888.220180] ret_from_fork+0x1f/0x30 [ 6892.056111] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 6892.061150] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6892.090741] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6892.096022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6892.099956] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6892.102762] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6892.109495] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6892.112046] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6892.115186] RDX: 0000000000000a89 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6892.117959] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6892.121365] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6892.123970] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6892.126474] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6892.130035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6892.132458] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6892.134491] Call Trace: [ 6892.135240] [ 6892.135799] ? show_regs.cold.9+0x22/0x2f [ 6892.136906] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6892.138660] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6892.140019] ? watchdog+0x40/0x40 [ 6892.140908] ? hrtimer_interrupt+0x128/0x2c0 [ 6892.142556] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6892.144215] ? apic_timer_interrupt+0xf/0x20 [ 6892.145713] [ 6892.146285] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6892.148129] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6892.149916] do_raw_spin_lock+0x177/0x190 [ 6892.151785] _raw_spin_lock+0x21/0x30 [ 6892.153412] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6892.155212] obd_mod_free+0x46/0x140 [ptlrpc] [ 6892.156989] mdc_commit_open+0x91/0xb0 [mdc] [ 6892.159622] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6892.161600] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6892.163785] after_reply+0xe2c/0x1780 [ptlrpc] [ 6892.165543] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6892.167839] ? __next_timer_interrupt+0x160/0x160 [ 6892.169613] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6892.171521] ? do_wait_intr+0xf0/0xf0 [ 6892.172723] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6892.174907] kthread+0x1d1/0x200 [ 6892.175995] ? set_kthread_struct+0x70/0x70 [ 6892.177459] ret_from_fork+0x1f/0x30 [ 6895.072354] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4381s! [ 6895.078725] Showing busy workqueues and worker pools: [ 6895.083245] workqueue events: flags=0x0 [ 6895.085190] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6895.088860] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6895.094621] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6895.099091] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6895.103061] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6895.109053] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6895.117360] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6895.120539] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6895.124980] workqueue events_long: flags=0x0 [ 6895.126892] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6895.129836] pending: lu_objects_flush [obdclass] [ 6895.132514] workqueue events_unbound: flags=0x2 [ 6895.134665] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6895.137279] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6895.140649] workqueue events_power_efficient: flags=0x80 [ 6895.143952] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6895.147204] pending: neigh_periodic_work, do_cache_clean [ 6895.150307] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6895.153700] pending: neigh_periodic_work, check_lifetime [ 6895.156109] workqueue mm_percpu_wq: flags=0x8 [ 6895.158353] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6895.161211] pending: vmstat_update, lru_add_drain_per_cpu [ 6895.163826] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6895.168184] pending: vmstat_update [ 6895.172154] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6895.177713] pending: lru_add_drain_per_cpu BAR(48) [ 6895.186737] workqueue ipv6_addrconf: flags=0x40008 [ 6895.189980] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6895.194213] pending: addrconf_verify_work [ 6895.197446] workqueue knbd0-recv: flags=0x1a [ 6895.201462] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6895.204616] in-flight: 650:recv_work [ 6895.206592] workqueue ptlrpc_pinger: flags=0x2 [ 6895.208641] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6895.211367] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6895.214074] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4382s workers=4 idle: 36 466 29757 [ 6895.218989] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6885s workers=2 idle: 652 [ 6901.070069] rcu: INFO: rcu_sched self-detected stall on CPU [ 6901.071620] rcu: 0-....: (3130328 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 6901.074131] (t=4920115 jiffies g=262441 q=58562) [ 6901.075267] rcu: rcu_sched kthread starved for 4389303 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 6901.078176] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6901.080425] rcu: RCU grace-period kthread stack dump: [ 6901.081575] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6901.083863] Call Trace: [ 6901.084545] __schedule+0x351/0xcb0 [ 6901.085601] schedule+0xc0/0x180 [ 6901.086477] schedule_timeout+0xb4/0x190 [ 6901.087453] ? do_raw_spin_unlock+0x75/0x190 [ 6901.088514] ? __next_timer_interrupt+0x160/0x160 [ 6901.091599] ? prepare_to_swait_event+0xf3/0x140 [ 6901.092670] rcu_gp_kthread+0x8dd/0x15b0 [ 6901.093576] ? force_qs_rnp+0x2c0/0x2c0 [ 6901.097707] kthread+0x1d1/0x200 [ 6901.099637] ? set_kthread_struct+0x70/0x70 [ 6901.101373] ret_from_fork+0x1f/0x30 [ 6901.102408] rcu: Stack dump where RCU GP kthread last ran: [ 6901.104322] Sending NMI from CPU 0 to CPUs 2: [ 6901.106885] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [ 6901.107848] NMI backtrace for cpu 0 [ 6901.115387] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6901.118604] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6901.120965] Call Trace: [ 6901.121927] [ 6901.122686] ? dump_stack+0xbb/0x10e [ 6901.123686] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6901.125566] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6901.130459] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6901.131947] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6901.134260] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6901.136864] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6901.138199] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6901.143984] ? update_process_times+0x6d/0xc0 [ 6901.147027] ? tick_sched_handle.isra.18+0x27/0x90 [ 6901.151371] ? tick_sched_timer+0x53/0xb0 [ 6901.153911] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6901.155861] ? tick_sched_handle.isra.18+0x90/0x90 [ 6901.157032] ? hrtimer_interrupt+0x128/0x2c0 [ 6901.162715] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6901.165203] ? apic_timer_interrupt+0xf/0x20 [ 6901.168563] [ 6901.169665] ? native_safe_halt+0x2e/0x50 [ 6901.171652] ? kvm_wait+0xcd/0xe0 [ 6901.174459] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6901.177909] ? do_raw_spin_lock+0x177/0x190 [ 6901.180685] ? _raw_spin_lock+0x21/0x30 [ 6901.182617] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6901.186506] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6901.189448] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6901.192260] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6901.195122] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6901.199494] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6901.201469] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6901.202957] ? __next_timer_interrupt+0x160/0x160 [ 6901.212024] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6901.213408] ? do_wait_intr+0xf0/0xf0 [ 6901.217873] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6901.219540] ? kthread+0x1d1/0x200 [ 6901.223937] ? set_kthread_struct+0x70/0x70 [ 6901.226603] ? ret_from_fork+0x1f/0x30 [ 6904.059968] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 6904.062087] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6904.064041] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel [ 6904.070730] Modules linked in: lustre(O) [ 6904.090006] ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6904.090033] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6904.090040] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6904.090044] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6904.090058] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6904.090062] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6904.090067] RAX: 000000000000548e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6904.090069] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6904.090071] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6904.090073] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6904.090075] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6904.090078] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6904.090081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6904.090084] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6904.090092] Call Trace: [ 6904.090097] [ 6904.090103] ? show_regs.cold.9+0x22/0x2f [ 6904.090112] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6904.090118] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6904.090122] ? watchdog+0x40/0x40 [ 6904.090127] ? hrtimer_interrupt+0x128/0x2c0 [ 6904.090130] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6904.090138] ? apic_timer_interrupt+0xf/0x20 [ 6904.090142] [ 6904.090145] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6904.090149] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6904.090157] do_raw_spin_lock+0x177/0x190 [ 6904.090162] _raw_spin_lock+0x21/0x30 [ 6904.090166] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6904.090689] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6904.091318] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6904.091676] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6904.092174] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6904.092684] ? woken_wake_function+0x30/0x30 [ 6904.092695] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6904.093018] kthread+0x1d1/0x200 [ 6904.093025] ? set_kthread_struct+0x70/0x70 [ 6904.093029] ret_from_fork+0x1f/0x30 [ 6904.247331] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6904.278291] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6904.293842] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6904.302648] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6904.307623] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6904.317079] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6904.320517] RAX: 0000000000001726 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6904.323996] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6904.327387] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6904.330678] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6904.334050] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6904.337429] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6904.341077] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6904.344238] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6904.347379] Call Trace: [ 6904.348620] [ 6904.349737] ? show_regs.cold.9+0x22/0x2f [ 6904.352132] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6904.354885] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6904.357202] ? watchdog+0x40/0x40 [ 6904.358872] ? hrtimer_interrupt+0x128/0x2c0 [ 6904.361360] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6904.363865] ? apic_timer_interrupt+0xf/0x20 [ 6904.366168] [ 6904.367163] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6904.369751] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6904.372486] do_raw_spin_lock+0x177/0x190 [ 6904.374510] _raw_spin_lock+0x21/0x30 [ 6904.376443] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6904.379576] ? do_raw_spin_unlock+0x75/0x190 [ 6904.381755] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6904.384489] ? do_wait_intr+0xf0/0xf0 [ 6904.386130] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6904.388958] mdc_reint+0x2d/0x110 [mdc] [ 6904.391700] mdc_unlink+0x658/0x870 [mdc] [ 6904.393853] lmv_unlink+0x34d/0xbd0 [lmv] [ 6904.396008] ll_unlink+0x24d/0x7f0 [lustre] [ 6904.398170] vfs_unlink+0x202/0x330 [ 6904.399719] do_unlinkat+0x2fa/0x460 [ 6904.401667] __x64_sys_unlink+0x37/0x50 [ 6904.403663] do_syscall_64+0xc1/0x440 [ 6904.405807] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6904.408747] RIP: 0033:0x7f83e5f5c10b [ 6904.410586] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6904.420125] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6904.423605] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6904.427496] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6904.430518] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6904.435540] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6904.440201] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6925.281071] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4412s! [ 6925.284590] Showing busy workqueues and worker pools: [ 6925.292041] workqueue events: flags=0x0 [ 6925.295981] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6925.300488] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6925.304658] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6925.310693] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6925.318284] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6925.322187] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6925.327616] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6925.331443] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6925.337557] workqueue events_long: flags=0x0 [ 6925.340281] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6925.343578] pending: lu_objects_flush [obdclass] [ 6925.346194] workqueue events_unbound: flags=0x2 [ 6925.348525] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6925.351388] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6925.354025] workqueue events_power_efficient: flags=0x80 [ 6925.356532] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6925.358768] pending: neigh_periodic_work, do_cache_clean [ 6925.362412] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6925.365710] pending: neigh_periodic_work, check_lifetime [ 6925.369293] workqueue mm_percpu_wq: flags=0x8 [ 6925.372913] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6925.376941] pending: vmstat_update, lru_add_drain_per_cpu [ 6925.379386] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6925.382510] pending: vmstat_update [ 6925.384164] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6925.387194] pending: lru_add_drain_per_cpu BAR(48) [ 6925.389532] workqueue ipv6_addrconf: flags=0x40008 [ 6925.392377] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6925.396095] pending: addrconf_verify_work [ 6925.397782] workqueue knbd0-recv: flags=0x1a [ 6925.399577] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6925.401430] in-flight: 650:recv_work [ 6925.403043] workqueue ptlrpc_pinger: flags=0x2 [ 6925.405054] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6925.409205] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6925.412079] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4412s workers=4 idle: 36 466 29757 [ 6925.416264] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6915s workers=2 idle: 652 [ 6928.060170] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 6928.065858] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6928.088206] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6928.094388] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6928.098850] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6928.103218] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6928.105228] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6928.113959] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6928.117511] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6928.121013] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6928.123912] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6928.127173] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6928.130070] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6928.133801] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6928.138042] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6928.141248] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6928.144129] Call Trace: [ 6928.145210] [ 6928.145973] ? show_regs.cold.9+0x22/0x2f [ 6928.147926] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6928.150030] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6928.151979] ? watchdog+0x40/0x40 [ 6928.153486] ? hrtimer_interrupt+0x128/0x2c0 [ 6928.155546] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6928.157802] ? apic_timer_interrupt+0xf/0x20 [ 6928.159600] [ 6928.160797] ? native_safe_halt+0x2e/0x50 [ 6928.162779] kvm_wait+0xcd/0xe0 [ 6928.164609] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6928.167596] do_raw_spin_lock+0x177/0x190 [ 6928.170062] _raw_spin_lock+0x21/0x30 [ 6928.171327] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6928.174262] process_one_work+0x2c8/0x700 [ 6928.176837] worker_thread+0x296/0x6e0 [ 6928.178897] ? rescuer_thread+0x570/0x570 [ 6928.182029] kthread+0x1d1/0x200 [ 6928.183582] ? set_kthread_struct+0x70/0x70 [ 6928.185934] ret_from_fork+0x1f/0x30 [ 6932.061096] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 6932.063483] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 6932.065947] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6932.066028] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6932.066036] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6932.066039] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6932.066054] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6932.066059] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6932.066063] RAX: 0000000000007ab3 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6932.066066] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6932.066068] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6932.066071] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6932.066073] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6932.066076] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6932.066079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6932.066081] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6932.066091] Call Trace: [ 6932.066096] [ 6932.066103] ? show_regs.cold.9+0x22/0x2f [ 6932.066112] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6932.066119] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6932.066123] ? watchdog+0x40/0x40 [ 6932.066127] ? hrtimer_interrupt+0x128/0x2c0 [ 6932.066131] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6932.066139] ? apic_timer_interrupt+0xf/0x20 [ 6932.066144] [ 6932.066147] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6932.066152] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6932.066158] do_raw_spin_lock+0x177/0x190 [ 6932.066163] _raw_spin_lock+0x21/0x30 [ 6932.066168] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6932.068197] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6932.068434] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6932.068662] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6932.068917] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6932.070308] ? woken_wake_function+0x30/0x30 [ 6932.070320] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6932.070556] kthread+0x1d1/0x200 [ 6932.070563] ? set_kthread_struct+0x70/0x70 [ 6932.070567] ret_from_fork+0x1f/0x30 [ 6932.245102] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6932.288745] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6932.299430] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6932.304819] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6932.312137] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6932.323380] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6932.327131] RAX: 0000000000006554 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6932.331139] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6932.335944] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6932.339973] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6932.345273] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6932.351287] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6932.360289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6932.365724] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6932.370189] Call Trace: [ 6932.371414] [ 6932.372550] ? show_regs.cold.9+0x22/0x2f [ 6932.375169] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6932.377550] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6932.380498] ? watchdog+0x40/0x40 [ 6932.382543] ? hrtimer_interrupt+0x128/0x2c0 [ 6932.388332] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6932.390973] ? apic_timer_interrupt+0xf/0x20 [ 6932.397757] [ 6932.399229] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6932.402479] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6932.406524] do_raw_spin_lock+0x177/0x190 [ 6932.410382] _raw_spin_lock+0x21/0x30 [ 6932.417126] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6932.423847] ? do_raw_spin_unlock+0x75/0x190 [ 6932.427146] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6932.430245] ? do_wait_intr+0xf0/0xf0 [ 6932.433083] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6932.436446] mdc_reint+0x2d/0x110 [mdc] [ 6932.438794] mdc_unlink+0x658/0x870 [mdc] [ 6932.441469] lmv_unlink+0x34d/0xbd0 [lmv] [ 6932.443835] ll_unlink+0x24d/0x7f0 [lustre] [ 6932.446475] vfs_unlink+0x202/0x330 [ 6932.452973] do_unlinkat+0x2fa/0x460 [ 6932.454892] __x64_sys_unlink+0x37/0x50 [ 6932.457388] do_syscall_64+0xc1/0x440 [ 6932.459462] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6932.462410] RIP: 0033:0x7f83e5f5c10b [ 6932.465302] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6932.484299] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6932.490200] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6932.497794] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6932.505076] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6932.512614] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6932.519241] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6952.056299] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 6952.059149] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6952.112552] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6952.118295] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6952.122568] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6952.128808] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6952.144998] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6952.153416] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6952.161786] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6952.167787] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6952.170295] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6952.177205] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6952.184863] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6952.196602] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6952.202263] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6952.206908] Call Trace: [ 6952.212271] [ 6952.215207] ? show_regs.cold.9+0x22/0x2f [ 6952.220625] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6952.223116] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6952.224604] ? watchdog+0x40/0x40 [ 6952.225932] ? hrtimer_interrupt+0x128/0x2c0 [ 6952.228300] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6952.230351] ? apic_timer_interrupt+0xf/0x20 [ 6952.231922] [ 6952.232569] ? native_safe_halt+0x2e/0x50 [ 6952.234128] kvm_wait+0xcd/0xe0 [ 6952.235413] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6952.237208] do_raw_spin_lock+0x177/0x190 [ 6952.242170] _raw_spin_lock+0x21/0x30 [ 6952.243087] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6952.244353] obd_mod_free+0x46/0x140 [ptlrpc] [ 6952.250855] mdc_commit_open+0x91/0xb0 [mdc] [ 6952.253681] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6952.256708] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6952.259395] after_reply+0xe2c/0x1780 [ptlrpc] [ 6952.262913] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6952.266195] ? __next_timer_interrupt+0x160/0x160 [ 6952.268226] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6952.269417] ? do_wait_intr+0xf0/0xf0 [ 6952.270263] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6952.273623] kthread+0x1d1/0x200 [ 6952.276168] ? set_kthread_struct+0x70/0x70 [ 6952.278269] ret_from_fork+0x1f/0x30 [ 6955.492404] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4442s! [ 6955.511141] Showing busy workqueues and worker pools: [ 6955.512860] workqueue events: flags=0x0 [ 6955.514181] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6955.526606] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6955.530618] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6955.533671] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6955.537963] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6955.541320] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6955.544926] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6955.548299] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6955.556880] workqueue events_long: flags=0x0 [ 6955.563118] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6955.565558] pending: lu_objects_flush [obdclass] [ 6955.570820] workqueue events_unbound: flags=0x2 [ 6955.579214] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6955.584844] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6955.588331] workqueue events_power_efficient: flags=0x80 [ 6955.594444] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6955.599883] pending: neigh_periodic_work, do_cache_clean [ 6955.601899] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6955.607428] pending: neigh_periodic_work, check_lifetime [ 6955.612335] workqueue mm_percpu_wq: flags=0x8 [ 6955.614386] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6955.618334] pending: vmstat_update, lru_add_drain_per_cpu [ 6955.621049] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6955.633122] pending: vmstat_update [ 6955.634785] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6955.638258] pending: lru_add_drain_per_cpu BAR(48) [ 6955.643566] workqueue ipv6_addrconf: flags=0x40008 [ 6955.650606] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6955.657936] pending: addrconf_verify_work [ 6955.659938] workqueue knbd0-recv: flags=0x1a [ 6955.665407] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6955.668262] in-flight: 650:recv_work [ 6955.673394] workqueue ptlrpc_pinger: flags=0x2 [ 6955.676368] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6955.680929] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6955.686623] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4442s workers=4 idle: 36 466 29757 [ 6955.693456] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6945s workers=2 idle: 652 [ 6956.064614] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 6956.071652] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6956.115906] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6956.123213] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6956.129098] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6956.136905] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6956.143387] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6956.161132] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6956.168062] RAX: 0000000000007e48 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6956.171363] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6956.174414] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6956.177540] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6956.181224] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6956.184495] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6956.191460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6956.197633] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6956.201649] Call Trace: [ 6956.203065] [ 6956.204027] ? show_regs.cold.9+0x22/0x2f [ 6956.205853] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6956.208246] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6956.210545] ? watchdog+0x40/0x40 [ 6956.212275] ? hrtimer_interrupt+0x128/0x2c0 [ 6956.214040] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6956.216293] ? apic_timer_interrupt+0xf/0x20 [ 6956.218125] [ 6956.218988] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6956.221022] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6956.223320] do_raw_spin_lock+0x177/0x190 [ 6956.225584] _raw_spin_lock+0x21/0x30 [ 6956.226842] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6956.228801] process_one_work+0x2c8/0x700 [ 6956.230315] worker_thread+0x296/0x6e0 [ 6956.232073] ? rescuer_thread+0x570/0x570 [ 6956.233813] kthread+0x1d1/0x200 [ 6956.235173] ? set_kthread_struct+0x70/0x70 [ 6956.236981] ret_from_fork+0x1f/0x30 [ 6960.059611] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 6960.063834] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6960.067475] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 6960.127618] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6960.127637] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6960.127640] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6960.127652] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6960.127655] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6960.127661] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6960.127663] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6960.127664] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6960.127666] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6960.127668] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6960.127671] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6960.127674] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6960.127675] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6960.127682] Call Trace: [ 6960.127686] [ 6960.127691] ? show_regs.cold.9+0x22/0x2f [ 6960.127699] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6960.127705] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6960.127708] ? watchdog+0x40/0x40 [ 6960.127711] ? hrtimer_interrupt+0x128/0x2c0 [ 6960.127715] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6960.127720] ? apic_timer_interrupt+0xf/0x20 [ 6960.127724] [ 6960.127726] ? native_safe_halt+0x2e/0x50 [ 6960.127730] kvm_wait+0xcd/0xe0 [ 6960.127736] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6960.127744] do_raw_spin_lock+0x177/0x190 [ 6960.127748] _raw_spin_lock+0x21/0x30 [ 6960.127751] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6960.128160] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6960.129866] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6960.130900] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6960.131819] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6960.133086] ? woken_wake_function+0x30/0x30 [ 6960.133097] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6960.134189] kthread+0x1d1/0x200 [ 6960.134196] ? set_kthread_struct+0x70/0x70 [ 6960.134200] ret_from_fork+0x1f/0x30 [ 6960.362619] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6960.413121] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6960.434789] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6960.444500] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6960.452041] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6960.462961] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 6960.471766] RAX: 0000000000003811 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6960.477418] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6960.481827] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6960.490250] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6960.498578] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6960.508269] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6960.515107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6960.524891] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6960.532720] Call Trace: [ 6960.535040] [ 6960.538327] ? show_regs.cold.9+0x22/0x2f [ 6960.542002] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6960.545202] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6960.549882] ? watchdog+0x40/0x40 [ 6960.552759] ? hrtimer_interrupt+0x128/0x2c0 [ 6960.554944] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6960.563797] ? apic_timer_interrupt+0xf/0x20 [ 6960.566191] [ 6960.570152] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6960.580859] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6960.584527] do_raw_spin_lock+0x177/0x190 [ 6960.591712] _raw_spin_lock+0x21/0x30 [ 6960.593867] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6960.599781] ? do_raw_spin_unlock+0x75/0x190 [ 6960.602174] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6960.605566] ? do_wait_intr+0xf0/0xf0 [ 6960.613492] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6960.616302] mdc_reint+0x2d/0x110 [mdc] [ 6960.618227] mdc_unlink+0x658/0x870 [mdc] [ 6960.620165] lmv_unlink+0x34d/0xbd0 [lmv] [ 6960.622157] ll_unlink+0x24d/0x7f0 [lustre] [ 6960.624775] vfs_unlink+0x202/0x330 [ 6960.627934] do_unlinkat+0x2fa/0x460 [ 6960.630713] __x64_sys_unlink+0x37/0x50 [ 6960.632897] do_syscall_64+0xc1/0x440 [ 6960.637081] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6960.640257] RIP: 0033:0x7f83e5f5c10b [ 6960.644224] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6960.661736] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6960.668262] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6960.671312] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6960.677612] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6960.684241] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6960.688499] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6980.056104] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 6980.066461] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6980.110590] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6980.118595] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6980.130758] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6980.139538] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6980.151518] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6980.157812] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 6980.162232] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 6980.168814] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 6980.173536] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 6980.177832] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6980.186596] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 6980.192393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6980.195676] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 6980.198782] Call Trace: [ 6980.199847] [ 6980.200829] ? show_regs.cold.9+0x22/0x2f [ 6980.203642] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6980.205393] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6980.207104] ? watchdog+0x40/0x40 [ 6980.208582] ? hrtimer_interrupt+0x128/0x2c0 [ 6980.210534] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6980.212710] ? apic_timer_interrupt+0xf/0x20 [ 6980.215359] [ 6980.216693] ? native_safe_halt+0x2e/0x50 [ 6980.218229] kvm_wait+0xcd/0xe0 [ 6980.218983] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6980.220623] do_raw_spin_lock+0x177/0x190 [ 6980.223356] _raw_spin_lock+0x21/0x30 [ 6980.225275] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6980.228690] obd_mod_free+0x46/0x140 [ptlrpc] [ 6980.231065] mdc_commit_open+0x91/0xb0 [mdc] [ 6980.232751] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6980.235271] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6980.237845] after_reply+0xe2c/0x1780 [ptlrpc] [ 6980.239870] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6980.241716] ? __next_timer_interrupt+0x160/0x160 [ 6980.243052] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6980.244567] ? do_wait_intr+0xf0/0xf0 [ 6980.246055] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6980.248395] kthread+0x1d1/0x200 [ 6980.249721] ? set_kthread_struct+0x70/0x70 [ 6980.251290] ret_from_fork+0x1f/0x30 [ 6984.061696] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 6984.067774] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6984.126914] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6984.134973] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6984.145795] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6984.149876] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6984.158183] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6984.175153] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6984.187072] RAX: 0000000000005284 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6984.197775] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 6984.207776] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 6984.214917] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 6984.220371] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 6984.225834] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 6984.232046] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6984.235216] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 6984.238960] Call Trace: [ 6984.240930] [ 6984.242073] ? show_regs.cold.9+0x22/0x2f [ 6984.248958] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6984.251928] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6984.262771] ? watchdog+0x40/0x40 [ 6984.267352] ? hrtimer_interrupt+0x128/0x2c0 [ 6984.274336] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6984.280904] ? apic_timer_interrupt+0xf/0x20 [ 6984.291722] [ 6984.296601] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6984.306837] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6984.308802] do_raw_spin_lock+0x177/0x190 [ 6984.310178] _raw_spin_lock+0x21/0x30 [ 6984.318162] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6984.323886] process_one_work+0x2c8/0x700 [ 6984.326577] worker_thread+0x296/0x6e0 [ 6984.329433] ? rescuer_thread+0x570/0x570 [ 6984.331839] kthread+0x1d1/0x200 [ 6984.334868] ? set_kthread_struct+0x70/0x70 [ 6984.345176] ret_from_fork+0x1f/0x30 [ 6986.209115] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4473s! [ 6986.216991] Showing busy workqueues and worker pools: [ 6986.222645] workqueue events: flags=0x0 [ 6986.228728] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6986.235152] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 6986.249969] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6986.265112] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 6986.274699] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6986.284757] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 6986.298092] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6986.310734] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 6986.322133] workqueue events_long: flags=0x0 [ 6986.326633] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6986.329989] pending: lu_objects_flush [obdclass] [ 6986.332765] workqueue events_unbound: flags=0x2 [ 6986.335605] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6986.340066] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6986.343272] workqueue events_power_efficient: flags=0x80 [ 6986.345642] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6986.355130] pending: neigh_periodic_work, do_cache_clean [ 6986.360001] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6986.370073] pending: neigh_periodic_work, check_lifetime [ 6986.377876] workqueue mm_percpu_wq: flags=0x8 [ 6986.384689] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6986.388750] pending: vmstat_update, lru_add_drain_per_cpu [ 6986.393043] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6986.399338] pending: vmstat_update [ 6986.404697] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 6986.412613] pending: lru_add_drain_per_cpu BAR(48) [ 6986.416987] workqueue ipv6_addrconf: flags=0x40008 [ 6986.432685] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6986.445051] pending: addrconf_verify_work [ 6986.450869] workqueue knbd0-recv: flags=0x1a [ 6986.453903] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6986.457618] in-flight: 650:recv_work [ 6986.460045] workqueue ptlrpc_pinger: flags=0x2 [ 6986.463603] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6986.468516] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 6986.470682] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4473s workers=4 idle: 36 466 29757 [ 6986.478642] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6976s workers=2 idle: 652 [ 6988.061587] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 6988.062026] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 6988.073762] Modules linked in: [ 6988.076487] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6988.076563] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6988.076570] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6988.076573] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6988.076587] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6988.076591] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6988.076595] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6988.076598] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 6988.076600] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6988.076602] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 6988.076604] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 6988.076607] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 6988.076610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6988.076612] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 6988.076621] Call Trace: [ 6988.076626] [ 6988.076632] ? show_regs.cold.9+0x22/0x2f [ 6988.076642] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6988.076649] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6988.076654] ? watchdog+0x40/0x40 [ 6988.076658] ? hrtimer_interrupt+0x128/0x2c0 [ 6988.076662] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6988.076669] ? apic_timer_interrupt+0xf/0x20 [ 6988.076673] [ 6988.076676] ? native_safe_halt+0x2e/0x50 [ 6988.076680] kvm_wait+0xcd/0xe0 [ 6988.076687] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6988.076694] do_raw_spin_lock+0x177/0x190 [ 6988.076698] _raw_spin_lock+0x21/0x30 [ 6988.076703] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6988.076972] ? do_raw_spin_unlock+0x75/0x190 [ 6988.076978] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6988.077236] ? do_wait_intr+0xf0/0xf0 [ 6988.077243] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6988.077432] mdc_reint+0x2d/0x110 [mdc] [ 6988.077468] mdc_unlink+0x658/0x870 [mdc] [ 6988.077491] lmv_unlink+0x34d/0xbd0 [lmv] [ 6988.077514] ll_unlink+0x24d/0x7f0 [lustre] [ 6988.077627] vfs_unlink+0x202/0x330 [ 6988.077634] do_unlinkat+0x2fa/0x460 [ 6988.077638] __x64_sys_unlink+0x37/0x50 [ 6988.077640] do_syscall_64+0xc1/0x440 [ 6988.077645] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6988.077649] RIP: 0033:0x7f83e5f5c10b [ 6988.077655] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 6988.077658] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 6988.077661] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 6988.077662] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 6988.077664] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 6988.077665] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 6988.077667] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 6988.399096] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6988.443215] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6988.452597] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6988.461084] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 6988.473727] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 6988.490720] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6988.499992] RAX: 0000000000004c47 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 6988.512075] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 6988.520762] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6988.527818] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 6988.535551] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 6988.542514] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 6988.561893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6988.563704] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 6988.575743] Call Trace: [ 6988.582622] [ 6988.586538] ? show_regs.cold.9+0x22/0x2f [ 6988.591012] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6988.603834] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6988.616189] ? watchdog+0x40/0x40 [ 6988.617293] ? hrtimer_interrupt+0x128/0x2c0 [ 6988.623411] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6988.627630] ? apic_timer_interrupt+0xf/0x20 [ 6988.630982] [ 6988.634172] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 6988.638754] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6988.644809] do_raw_spin_lock+0x177/0x190 [ 6988.646197] _raw_spin_lock+0x21/0x30 [ 6988.652937] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6988.661692] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6988.672699] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6988.682585] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6988.686217] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6988.690133] ? woken_wake_function+0x30/0x30 [ 6988.692525] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6988.700797] kthread+0x1d1/0x200 [ 6988.702510] ? set_kthread_struct+0x70/0x70 [ 6988.707110] ret_from_fork+0x1f/0x30 [ 7012.056071] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 7012.060805] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7012.061507] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 7012.108548] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7012.108563] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7012.108566] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7012.108580] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7012.108585] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7012.108591] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7012.108594] RDX: 0000000000002631 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7012.108596] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7012.108598] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7012.108601] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7012.108604] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7012.108607] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7012.108610] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7012.108618] Call Trace: [ 7012.108623] [ 7012.108629] ? show_regs.cold.9+0x22/0x2f [ 7012.112649] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7012.112729] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7012.112735] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7012.112742] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7012.113226] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7012.113237] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7012.113241] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7012.113246] RAX: 000000000000328e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7012.113249] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7012.113251] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7012.113253] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7012.113256] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7012.113259] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7012.113262] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7012.113264] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7012.113273] Call Trace: [ 7012.113278] [ 7012.113284] ? show_regs.cold.9+0x22/0x2f [ 7012.113294] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7012.113300] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7012.113305] ? watchdog+0x40/0x40 [ 7012.113309] ? hrtimer_interrupt+0x128/0x2c0 [ 7012.113313] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7012.113322] ? apic_timer_interrupt+0xf/0x20 [ 7012.113327] [ 7012.113330] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7012.113569] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7012.113575] do_raw_spin_lock+0x177/0x190 [ 7012.113580] _raw_spin_lock+0x21/0x30 [ 7012.113585] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7012.113821] process_one_work+0x2c8/0x700 [ 7012.113828] worker_thread+0x296/0x6e0 [ 7012.113832] ? rescuer_thread+0x570/0x570 [ 7012.113836] kthread+0x1d1/0x200 [ 7012.113841] ? set_kthread_struct+0x70/0x70 [ 7012.113845] ret_from_fork+0x1f/0x30 [ 7012.510864] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7012.513572] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7012.515918] ? watchdog+0x40/0x40 [ 7012.517805] ? hrtimer_interrupt+0x128/0x2c0 [ 7012.521090] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7012.531242] ? apic_timer_interrupt+0xf/0x20 [ 7012.535111] [ 7012.536620] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7012.544447] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7012.547918] do_raw_spin_lock+0x177/0x190 [ 7012.565791] _raw_spin_lock+0x21/0x30 [ 7012.576518] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7012.580290] obd_mod_free+0x46/0x140 [ptlrpc] [ 7012.583338] mdc_commit_open+0x91/0xb0 [mdc] [ 7012.586537] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7012.591534] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7012.595276] after_reply+0xe2c/0x1780 [ptlrpc] [ 7012.599527] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7012.604516] ? __next_timer_interrupt+0x160/0x160 [ 7012.609551] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7012.616862] ? do_wait_intr+0xf0/0xf0 [ 7012.617879] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7012.629302] kthread+0x1d1/0x200 [ 7012.635317] ? set_kthread_struct+0x70/0x70 [ 7012.641627] ret_from_fork+0x1f/0x30 [ 7016.062018] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 7016.064661] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 7016.065301] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7016.065372] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7016.065379] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7016.065382] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7016.065398] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7016.065402] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7016.065406] RAX: 0000000000004629 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7016.065408] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7016.065411] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7016.065413] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7016.065415] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7016.065418] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7016.065421] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7016.065422] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7016.065432] Call Trace: [ 7016.065437] [ 7016.065443] ? show_regs.cold.9+0x22/0x2f [ 7016.065452] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7016.065457] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7016.065461] ? watchdog+0x40/0x40 [ 7016.065465] ? hrtimer_interrupt+0x128/0x2c0 [ 7016.065469] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7016.065478] ? apic_timer_interrupt+0xf/0x20 [ 7016.065482] [ 7016.065485] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7016.065490] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7016.065495] do_raw_spin_lock+0x177/0x190 [ 7016.065500] _raw_spin_lock+0x21/0x30 [ 7016.065506] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7016.065992] ? do_raw_spin_unlock+0x75/0x190 [ 7016.065998] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7016.066193] ? do_wait_intr+0xf0/0xf0 [ 7016.066200] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7016.066389] mdc_reint+0x2d/0x110 [mdc] [ 7016.066428] mdc_unlink+0x658/0x870 [mdc] [ 7016.066455] lmv_unlink+0x34d/0xbd0 [lmv] [ 7016.066485] ll_unlink+0x24d/0x7f0 [lustre] [ 7016.066710] vfs_unlink+0x202/0x330 [ 7016.066719] do_unlinkat+0x2fa/0x460 [ 7016.066833] __x64_sys_unlink+0x37/0x50 [ 7016.066836] do_syscall_64+0xc1/0x440 [ 7016.066842] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7016.066848] RIP: 0033:0x7f83e5f5c10b [ 7016.066854] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7016.066858] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7016.066863] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7016.066865] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7016.066868] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7016.066870] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7016.066872] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7016.237641] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7016.258270] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7016.263029] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7016.267074] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7016.269912] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7016.279015] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7016.282659] RAX: 0000000000002b0d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7016.286319] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7016.289830] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7016.293068] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7016.296399] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7016.299466] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7016.303042] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7016.305196] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7016.308480] Call Trace: [ 7016.309868] [ 7016.310626] ? show_regs.cold.9+0x22/0x2f [ 7016.312418] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7016.314798] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7016.316997] ? watchdog+0x40/0x40 [ 7016.318515] ? hrtimer_interrupt+0x128/0x2c0 [ 7016.320432] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7016.323222] ? apic_timer_interrupt+0xf/0x20 [ 7016.325192] [ 7016.326528] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7016.329012] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7016.331343] do_raw_spin_lock+0x177/0x190 [ 7016.333302] _raw_spin_lock+0x21/0x30 [ 7016.334810] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7016.337836] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7016.340343] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7016.342787] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7016.345587] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7016.347969] ? woken_wake_function+0x30/0x30 [ 7016.349555] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7016.352245] kthread+0x1d1/0x200 [ 7016.353630] ? set_kthread_struct+0x70/0x70 [ 7016.355522] ret_from_fork+0x1f/0x30 [ 7016.929031] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4503s! [ 7016.932977] Showing busy workqueues and worker pools: [ 7016.935398] workqueue events: flags=0x0 [ 7016.937118] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7016.939911] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7016.943304] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7016.946669] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7016.949609] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7016.953221] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7016.957308] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7016.960555] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7016.964614] workqueue events_long: flags=0x0 [ 7016.966822] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7016.970107] pending: lu_objects_flush [obdclass] [ 7016.973117] workqueue events_unbound: flags=0x2 [ 7016.974790] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7016.977813] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7016.980292] workqueue events_power_efficient: flags=0x80 [ 7016.984275] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7016.987392] pending: neigh_periodic_work, do_cache_clean [ 7016.989758] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7016.992664] pending: neigh_periodic_work, check_lifetime [ 7016.995847] workqueue mm_percpu_wq: flags=0x8 [ 7016.997910] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7017.000707] pending: vmstat_update, lru_add_drain_per_cpu [ 7017.003414] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7017.005892] pending: vmstat_update [ 7017.007477] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7017.010745] pending: lru_add_drain_per_cpu BAR(48) [ 7017.013480] workqueue ipv6_addrconf: flags=0x40008 [ 7017.016205] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7017.019896] pending: addrconf_verify_work [ 7017.022365] workqueue knbd0-recv: flags=0x1a [ 7017.023753] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7017.028015] in-flight: 650:recv_work [ 7017.029266] workqueue ptlrpc_pinger: flags=0x2 [ 7017.031524] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7017.034823] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7017.036868] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4503s workers=4 idle: 36 466 29757 [ 7017.041261] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7007s workers=2 idle: 652 [ 7040.066030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 7040.078572] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7040.162838] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7040.172830] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7040.180299] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7040.186631] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7040.195608] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7040.214050] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7040.220486] RAX: 0000000000007e27 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7040.224132] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7040.226778] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7040.230327] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7040.235568] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7040.239516] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7040.245344] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7040.249263] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7040.253123] Call Trace: [ 7040.254243] [ 7040.256200] ? show_regs.cold.9+0x22/0x2f [ 7040.260908] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7040.263124] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7040.266053] ? watchdog+0x40/0x40 [ 7040.270785] ? hrtimer_interrupt+0x128/0x2c0 [ 7040.275805] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7040.278159] ? apic_timer_interrupt+0xf/0x20 [ 7040.286032] [ 7040.287113] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7040.292164] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7040.294578] do_raw_spin_lock+0x177/0x190 [ 7040.299785] _raw_spin_lock+0x21/0x30 [ 7040.301473] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7040.305466] process_one_work+0x2c8/0x700 [ 7040.308012] worker_thread+0x296/0x6e0 [ 7040.309390] ? rescuer_thread+0x570/0x570 [ 7040.311067] kthread+0x1d1/0x200 [ 7040.312414] ? set_kthread_struct+0x70/0x70 [ 7040.313930] ret_from_fork+0x1f/0x30 [ 7044.056119] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 7044.059029] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 7044.059732] Modules linked in: [ 7044.062028] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 7044.062040] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7044.062108] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7044.062114] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7044.062117] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7044.062129] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7044.062133] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7044.062137] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7044.062140] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7044.062142] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7044.062144] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7044.062147] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7044.062150] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7044.062153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7044.062155] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7044.062164] Call Trace: [ 7044.062170] [ 7044.062184] ? show_regs.cold.9+0x22/0x2f [ 7044.062195] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7044.062201] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7044.062206] ? watchdog+0x40/0x40 [ 7044.062210] ? hrtimer_interrupt+0x128/0x2c0 [ 7044.062214] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7044.062222] ? apic_timer_interrupt+0xf/0x20 [ 7044.062226] [ 7044.062229] ? native_safe_halt+0x2e/0x50 [ 7044.062233] kvm_wait+0xcd/0xe0 [ 7044.062241] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7044.062248] do_raw_spin_lock+0x177/0x190 [ 7044.062252] _raw_spin_lock+0x21/0x30 [ 7044.062257] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7044.062532] ? do_raw_spin_unlock+0x75/0x190 [ 7044.062538] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7044.062750] ? do_wait_intr+0xf0/0xf0 [ 7044.062758] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7044.062893] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7044.062953] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7044.062958] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7044.062963] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7044.062973] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7044.062977] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7044.062982] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7044.062984] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7044.062987] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7044.062989] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7044.062966] mdc_reint+0x2d/0x110 [mdc] [ 7044.062991] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7044.062996] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7044.062999] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7044.063001] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7044.063008] Call Trace: [ 7044.063014] [ 7044.063009] mdc_unlink+0x658/0x870 [mdc] [ 7044.063017] ? show_regs.cold.9+0x22/0x2f [ 7044.063029] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7044.063038] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7044.063043] ? watchdog+0x40/0x40 [ 7044.063048] ? hrtimer_interrupt+0x128/0x2c0 [ 7044.063036] lmv_unlink+0x34d/0xbd0 [lmv] [ 7044.063052] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7044.063060] ? apic_timer_interrupt+0xf/0x20 [ 7044.063069] [ 7044.063073] ? native_safe_halt+0x2e/0x50 [ 7044.063078] kvm_wait+0xcd/0xe0 [ 7044.063085] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7044.063092] do_raw_spin_lock+0x177/0x190 [ 7044.063096] _raw_spin_lock+0x21/0x30 [ 7044.063072] ll_unlink+0x24d/0x7f0 [lustre] [ 7044.063636] vfs_unlink+0x202/0x330 [ 7044.063644] do_unlinkat+0x2fa/0x460 [ 7044.063653] __x64_sys_unlink+0x37/0x50 [ 7044.063657] do_syscall_64+0xc1/0x440 [ 7044.063663] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7044.063669] RIP: 0033:0x7f83e5f5c10b [ 7044.063100] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7044.063675] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7044.063680] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7044.063685] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7044.063687] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7044.063689] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7044.063695] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7044.063697] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7044.063744] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7044.063869] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7044.064036] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7044.064179] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7044.064352] ? woken_wake_function+0x30/0x30 [ 7044.064360] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7044.064568] kthread+0x1d1/0x200 [ 7044.064572] ? set_kthread_struct+0x70/0x70 [ 7044.064575] ret_from_fork+0x1f/0x30 [ 7044.502231] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7044.528200] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7044.534224] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7044.538782] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7044.544233] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7044.562597] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7044.589911] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7044.601064] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7044.612363] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7044.621306] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7044.625569] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7044.635026] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7044.637981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7044.643421] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7044.649130] Call Trace: [ 7044.651128] [ 7044.652624] ? show_regs.cold.9+0x22/0x2f [ 7044.655664] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7044.662206] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7044.669687] ? watchdog+0x40/0x40 [ 7044.676721] ? hrtimer_interrupt+0x128/0x2c0 [ 7044.688315] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7044.689957] ? apic_timer_interrupt+0xf/0x20 [ 7044.700759] [ 7044.704858] ? native_safe_halt+0x2e/0x50 [ 7044.706230] kvm_wait+0xcd/0xe0 [ 7044.718770] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7044.726255] do_raw_spin_lock+0x177/0x190 [ 7044.729145] _raw_spin_lock+0x21/0x30 [ 7044.735779] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7044.746558] obd_mod_free+0x46/0x140 [ptlrpc] [ 7044.751665] mdc_commit_open+0x91/0xb0 [mdc] [ 7044.757518] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7044.762712] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7044.774621] after_reply+0xe2c/0x1780 [ptlrpc] [ 7044.782566] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7044.791716] ? __next_timer_interrupt+0x160/0x160 [ 7044.801341] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7044.813025] ? do_wait_intr+0xf0/0xf0 [ 7044.818285] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7044.826891] kthread+0x1d1/0x200 [ 7044.832258] ? set_kthread_struct+0x70/0x70 [ 7044.840861] ret_from_fork+0x1f/0x30 [ 7047.137517] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4533s! [ 7047.142321] Showing busy workqueues and worker pools: [ 7047.146461] workqueue events: flags=0x0 [ 7047.148493] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7047.152128] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7047.167492] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7047.178386] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7047.192368] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7047.205763] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7047.218398] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7047.226504] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7047.232635] workqueue events_long: flags=0x0 [ 7047.234506] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7047.238117] pending: lu_objects_flush [obdclass] [ 7047.243463] workqueue events_unbound: flags=0x2 [ 7047.250728] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7047.255951] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7047.263654] workqueue events_power_efficient: flags=0x80 [ 7047.271314] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7047.276471] pending: neigh_periodic_work, do_cache_clean [ 7047.280441] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7047.290553] pending: neigh_periodic_work, check_lifetime [ 7047.295985] workqueue mm_percpu_wq: flags=0x8 [ 7047.298976] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7047.306028] pending: vmstat_update, lru_add_drain_per_cpu [ 7047.318809] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7047.323896] pending: vmstat_update [ 7047.326185] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7047.338817] pending: lru_add_drain_per_cpu BAR(48) [ 7047.347828] workqueue ipv6_addrconf: flags=0x40008 [ 7047.350046] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7047.353404] pending: addrconf_verify_work [ 7047.355860] workqueue knbd0-recv: flags=0x1a [ 7047.358605] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7047.369393] in-flight: 650:recv_work [ 7047.371291] workqueue ptlrpc_pinger: flags=0x2 [ 7047.373601] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7047.381928] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7047.384565] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4534s workers=4 idle: 36 466 29757 [ 7047.388663] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7037s workers=2 idle: 652 [ 7068.060965] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 7068.070246] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7068.111542] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7068.116809] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7068.121510] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7068.124490] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7068.127268] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7068.136218] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7068.139476] RAX: 0000000000007ec1 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7068.142550] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7068.145734] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7068.148600] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7068.151057] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7068.154794] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7068.158817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7068.161784] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7068.164929] Call Trace: [ 7068.166063] [ 7068.168075] ? show_regs.cold.9+0x22/0x2f [ 7068.169883] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7068.172056] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7068.173888] ? watchdog+0x40/0x40 [ 7068.175538] ? hrtimer_interrupt+0x128/0x2c0 [ 7068.177356] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7068.179514] ? apic_timer_interrupt+0xf/0x20 [ 7068.181484] [ 7068.182561] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7068.184899] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7068.187495] do_raw_spin_lock+0x177/0x190 [ 7068.189750] _raw_spin_lock+0x21/0x30 [ 7068.192127] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7068.194963] process_one_work+0x2c8/0x700 [ 7068.196772] worker_thread+0x296/0x6e0 [ 7068.198691] ? rescuer_thread+0x570/0x570 [ 7068.200537] kthread+0x1d1/0x200 [ 7068.202114] ? set_kthread_struct+0x70/0x70 [ 7068.204119] ret_from_fork+0x1f/0x30 [ 7072.056091] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 7072.060086] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7072.061089] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 7072.062014] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7072.062021] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7072.062066] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7072.062070] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7072.062073] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7072.062086] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7072.062090] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7072.062093] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7072.062095] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7072.062096] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7072.062097] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7072.062099] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7072.062102] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7072.062104] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7072.062105] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7072.062147] Call Trace: [ 7072.062153] [ 7072.062157] ? show_regs.cold.9+0x22/0x2f [ 7072.062165] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7072.062170] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7072.062391] ? watchdog+0x40/0x40 [ 7072.062396] ? hrtimer_interrupt+0x128/0x2c0 [ 7072.062400] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7072.062406] ? apic_timer_interrupt+0xf/0x20 [ 7072.062410] [ 7072.062413] ? native_safe_halt+0x2e/0x50 [ 7072.062417] kvm_wait+0xcd/0xe0 [ 7072.062424] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7072.062430] do_raw_spin_lock+0x177/0x190 [ 7072.062434] _raw_spin_lock+0x21/0x30 [ 7072.062440] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7072.062710] ? do_raw_spin_unlock+0x75/0x190 [ 7072.062716] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7072.062917] ? do_wait_intr+0xf0/0xf0 [ 7072.062924] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7072.063097] mdc_reint+0x2d/0x110 [mdc] [ 7072.063127] mdc_unlink+0x658/0x870 [mdc] [ 7072.063143] lmv_unlink+0x34d/0xbd0 [lmv] [ 7072.063162] ll_unlink+0x24d/0x7f0 [lustre] [ 7072.063564] vfs_unlink+0x202/0x330 [ 7072.063570] do_unlinkat+0x2fa/0x460 [ 7072.063575] __x64_sys_unlink+0x37/0x50 [ 7072.063578] do_syscall_64+0xc1/0x440 [ 7072.063583] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7072.063588] RIP: 0033:0x7f83e5f5c10b [ 7072.063594] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7072.063597] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7072.063601] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7072.063603] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7072.063605] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7072.063608] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7072.063610] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7072.117203] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7072.137821] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod [ 7072.141113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7072.141123] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7072.141138] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7072.141142] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7072.141148] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7072.141150] RDX: 0000000000007f80 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7072.141153] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7072.141156] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7072.141158] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7072.141162] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7072.141165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7072.141167] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7072.141175] Call Trace: [ 7072.141179] [ 7072.141185] ? show_regs.cold.9+0x22/0x2f [ 7072.141194] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7072.141200] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7072.141204] ? watchdog+0x40/0x40 [ 7072.141386] ? hrtimer_interrupt+0x128/0x2c0 [ 7072.141392] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7072.141400] ? apic_timer_interrupt+0xf/0x20 [ 7072.141405] [ 7072.141408] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7072.141413] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7072.141418] do_raw_spin_lock+0x177/0x190 [ 7072.141422] _raw_spin_lock+0x21/0x30 [ 7072.141429] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7072.143883] obd_mod_free+0x46/0x140 [ptlrpc] [ 7072.144092] mdc_commit_open+0x91/0xb0 [mdc] [ 7072.144130] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7072.144332] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7072.144555] after_reply+0xe2c/0x1780 [ptlrpc] [ 7072.194458] sha512_ssse3 sha512_generic [ 7072.194473] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7072.194480] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7072.194482] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7072.194495] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7072.194499] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7072.194503] RAX: 00000000000063ea RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7072.194505] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7072.194507] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7072.194509] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7072.194511] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7072.194514] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7072.194517] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7072.194519] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7072.194527] Call Trace: [ 7072.194533] [ 7072.194540] ? show_regs.cold.9+0x22/0x2f [ 7072.194549] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7072.194556] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7072.194560] ? watchdog+0x40/0x40 [ 7072.194565] ? hrtimer_interrupt+0x128/0x2c0 [ 7072.194569] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7072.194577] ? apic_timer_interrupt+0xf/0x20 [ 7072.194581] [ 7072.194584] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7072.194589] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7072.194608] do_raw_spin_lock+0x177/0x190 [ 7072.194613] _raw_spin_lock+0x21/0x30 [ 7072.194618] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7072.195082] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7072.195277] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7072.195512] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7072.195738] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7072.197465] ? woken_wake_function+0x30/0x30 [ 7072.197477] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7072.215936] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7072.220422] kthread+0x1d1/0x200 [ 7072.220442] ? set_kthread_struct+0x70/0x70 [ 7072.220446] ret_from_fork+0x1f/0x30 [ 7072.932963] ? __next_timer_interrupt+0x160/0x160 [ 7072.938415] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7072.944798] ? do_wait_intr+0xf0/0xf0 [ 7072.946096] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7072.950488] kthread+0x1d1/0x200 [ 7072.953519] ? set_kthread_struct+0x70/0x70 [ 7072.962066] ret_from_fork+0x1f/0x30 [ 7077.862032] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4564s! [ 7077.872671] Showing busy workqueues and worker pools: [ 7077.875658] workqueue events: flags=0x0 [ 7077.877853] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7077.881888] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7077.890711] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7077.893455] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7077.898487] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7077.901790] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7077.908693] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7077.912620] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7077.918210] workqueue events_long: flags=0x0 [ 7077.920232] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7077.925538] pending: lu_objects_flush [obdclass] [ 7077.928247] workqueue events_unbound: flags=0x2 [ 7077.930033] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7077.933064] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7077.938164] workqueue events_power_efficient: flags=0x80 [ 7077.945124] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7077.949190] pending: neigh_periodic_work, do_cache_clean [ 7077.954316] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7077.957553] pending: neigh_periodic_work, check_lifetime [ 7077.960305] workqueue mm_percpu_wq: flags=0x8 [ 7077.963071] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7077.965948] pending: vmstat_update, lru_add_drain_per_cpu [ 7077.968114] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7077.973479] pending: vmstat_update [ 7077.977033] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7077.981440] pending: lru_add_drain_per_cpu BAR(48) [ 7077.986098] workqueue ipv6_addrconf: flags=0x40008 [ 7077.988155] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7077.992450] pending: addrconf_verify_work [ 7077.997891] workqueue knbd0-recv: flags=0x1a [ 7078.000744] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7078.005893] in-flight: 650:recv_work [ 7078.007647] workqueue ptlrpc_pinger: flags=0x2 [ 7078.011472] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7078.015730] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7078.020712] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4564s workers=4 idle: 36 466 29757 [ 7078.025910] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7068s workers=2 idle: 652 [ 7081.076343] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 7081.086998] rcu: 0-....: (3240893 ticks this GP) idle=30a/1/0x4000000000000000 softirq=128519/128519 fqs=126395 [ 7081.098906] (detected by 1, t=5100117 jiffies, g=262441, q=58652) [ 7081.101167] Sending NMI from CPU 1 to CPUs 0: [ 7081.124508] NMI backtrace for cpu 0 [ 7081.124517] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7081.124519] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7081.124521] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7081.124523] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7081.124525] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 [ 7081.124528] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7081.124529] RDX: 0000000000006f41 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7081.124530] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7081.124531] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7081.124533] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7081.124534] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7081.124536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7081.124538] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7081.124539] Call Trace: [ 7081.124540] [ 7081.124541] ? show_regs.cold.9+0x22/0x2f [ 7081.124542] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7081.124544] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7081.124545] ? nmi_handle+0xbd/0x210 [ 7081.124546] ? default_do_nmi+0x4b/0x1b0 [ 7081.124547] ? do_nmi+0x280/0x380 [ 7081.124548] ? end_repeat_nmi+0x16/0x1f [ 7081.124549] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7081.124550] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7081.124552] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7081.124553] [ 7081.124554] do_raw_spin_lock+0x177/0x190 [ 7081.124555] _raw_spin_lock+0x21/0x30 [ 7081.124556] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7081.124557] obd_mod_free+0x46/0x140 [ptlrpc] [ 7081.124559] mdc_commit_open+0x91/0xb0 [mdc] [ 7081.124560] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7081.124561] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7081.124562] after_reply+0xe2c/0x1780 [ptlrpc] [ 7081.124563] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7081.124564] ? __next_timer_interrupt+0x160/0x160 [ 7081.124565] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7081.124566] ? do_wait_intr+0xf0/0xf0 [ 7081.124567] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7081.124568] kthread+0x1d1/0x200 [ 7081.124569] ? set_kthread_struct+0x70/0x70 [ 7081.124570] ret_from_fork+0x1f/0x30 [ 7081.130272] rcu: rcu_sched kthread starved for 4569304 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 7081.517450] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7081.532741] rcu: RCU grace-period kthread stack dump: [ 7081.548895] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7081.566157] Call Trace: [ 7081.573696] __schedule+0x351/0xcb0 [ 7081.575213] schedule+0xc0/0x180 [ 7081.580056] schedule_timeout+0xb4/0x190 [ 7081.585712] ? do_raw_spin_unlock+0x75/0x190 [ 7081.587245] ? __next_timer_interrupt+0x160/0x160 [ 7081.604304] ? prepare_to_swait_event+0xf3/0x140 [ 7081.614618] rcu_gp_kthread+0x8dd/0x15b0 [ 7081.615974] ? force_qs_rnp+0x2c0/0x2c0 [ 7081.617971] kthread+0x1d1/0x200 [ 7081.620664] ? set_kthread_struct+0x70/0x70 [ 7081.633750] ret_from_fork+0x1f/0x30 [ 7081.636812] rcu: Stack dump where RCU GP kthread last ran: [ 7081.644075] Sending NMI from CPU 1 to CPUs 2: [ 7081.648413] NMI backtrace for cpu 2 [ 7081.648419] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7081.648422] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7081.648424] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7081.648426] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7081.648428] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7081.648430] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 [ 7081.648431] RAX: 0000000000007e86 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7081.648433] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7081.648434] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7081.648436] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7081.648437] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7081.648438] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7081.648440] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7081.648441] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7081.648442] Call Trace: [ 7081.648444] [ 7081.648444] ? show_regs.cold.9+0x22/0x2f [ 7081.648446] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7081.648447] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7081.648448] ? nmi_handle+0xbd/0x210 [ 7081.648449] ? default_do_nmi+0x4b/0x1b0 [ 7081.648450] ? do_nmi+0x280/0x380 [ 7081.648452] ? end_repeat_nmi+0x16/0x1f [ 7081.648453] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7081.648454] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7081.648456] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7081.648457] [ 7081.648458] do_raw_spin_lock+0x177/0x190 [ 7081.648459] _raw_spin_lock+0x21/0x30 [ 7081.648460] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7081.648461] process_one_work+0x2c8/0x700 [ 7081.648462] worker_thread+0x296/0x6e0 [ 7081.648464] ? rescuer_thread+0x570/0x570 [ 7081.648464] kthread+0x1d1/0x200 [ 7081.648465] ? set_kthread_struct+0x70/0x70 [ 7081.648466] ret_from_fork+0x1f/0x30 [ 7100.062043] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7100.067806] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7100.110446] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7100.124920] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7100.136186] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7100.142634] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7100.153712] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7100.160829] RAX: 0000000000002e7c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7100.167785] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7100.175690] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7100.181102] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7100.183777] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7100.193473] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7100.201517] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7100.211653] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7100.219417] Call Trace: [ 7100.220652] [ 7100.222665] ? show_regs.cold.9+0x22/0x2f [ 7100.227601] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7100.235929] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7100.243787] ? watchdog+0x40/0x40 [ 7100.250457] ? hrtimer_interrupt+0x128/0x2c0 [ 7100.254181] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7100.258416] ? apic_timer_interrupt+0xf/0x20 [ 7100.263674] [ 7100.264665] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7100.274836] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7100.283443] do_raw_spin_lock+0x177/0x190 [ 7100.286623] _raw_spin_lock+0x21/0x30 [ 7100.291214] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7100.296920] ? do_raw_spin_unlock+0x75/0x190 [ 7100.308511] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7100.317733] ? do_wait_intr+0xf0/0xf0 [ 7100.322606] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7100.330858] mdc_reint+0x2d/0x110 [mdc] [ 7100.337625] mdc_unlink+0x658/0x870 [mdc] [ 7100.340778] lmv_unlink+0x34d/0xbd0 [lmv] [ 7100.346195] ll_unlink+0x24d/0x7f0 [lustre] [ 7100.348878] vfs_unlink+0x202/0x330 [ 7100.354061] do_unlinkat+0x2fa/0x460 [ 7100.359621] __x64_sys_unlink+0x37/0x50 [ 7100.363236] do_syscall_64+0xc1/0x440 [ 7100.365193] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7100.367663] RIP: 0033:0x7f83e5f5c10b [ 7100.369723] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7100.379149] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7100.382001] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7100.385678] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7100.389578] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7100.392186] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7100.395174] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7108.058952] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 7108.059070] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 7108.064805] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 7108.064820] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7108.065102] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7108.065106] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7108.065112] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7108.065455] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7108.065466] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7108.065470] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7108.065475] RAX: 000000000000041c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7108.065477] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7108.065479] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7108.065482] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7108.065484] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7108.065488] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7108.065491] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7108.065493] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7108.065503] Call Trace: [ 7108.065508] [ 7108.065514] ? show_regs.cold.9+0x22/0x2f [ 7108.065525] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7108.065532] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7108.065535] ? watchdog+0x40/0x40 [ 7108.065540] ? hrtimer_interrupt+0x128/0x2c0 [ 7108.065544] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7108.065553] ? apic_timer_interrupt+0xf/0x20 [ 7108.065557] [ 7108.065561] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7108.065566] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7108.065571] do_raw_spin_lock+0x177/0x190 [ 7108.065576] _raw_spin_lock+0x21/0x30 [ 7108.065582] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7108.065792] process_one_work+0x2c8/0x700 [ 7108.065799] worker_thread+0x296/0x6e0 [ 7108.065804] ? rescuer_thread+0x570/0x570 [ 7108.065807] kthread+0x1d1/0x200 [ 7108.065813] ? set_kthread_struct+0x70/0x70 [ 7108.065817] ret_from_fork+0x1f/0x30 [ 7108.068969] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [ 7108.071824] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7108.071906] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7108.071913] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7108.071916] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7108.071938] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7108.071943] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7108.071947] RAX: 0000000000002983 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7108.071949] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7108.071952] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7108.071954] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7108.071956] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7108.071959] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7108.071966] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7108.071968] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7108.071975] Call Trace: [ 7108.071982] [ 7108.071992] ? show_regs.cold.9+0x22/0x2f [ 7108.072014] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7108.072022] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7108.072027] ? watchdog+0x40/0x40 [ 7108.072031] ? hrtimer_interrupt+0x128/0x2c0 [ 7108.072036] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7108.072044] ? apic_timer_interrupt+0xf/0x20 [ 7108.072049] [ 7108.072057] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7108.072062] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7108.072068] do_raw_spin_lock+0x177/0x190 [ 7108.072073] _raw_spin_lock+0x21/0x30 [ 7108.072078] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7108.076539] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) [ 7108.109553] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7108.109876] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7108.110100] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7108.110320] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7108.112842] ? woken_wake_function+0x30/0x30 [ 7108.112853] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7108.113105] kthread+0x1d1/0x200 [ 7108.113112] ? set_kthread_struct+0x70/0x70 [ 7108.113116] ret_from_fork+0x1f/0x30 [ 7108.502516] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7108.522369] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7108.529022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7108.534305] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7108.541533] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7108.549783] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7108.553477] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7108.557646] RDX: 0000000000007d70 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7108.565548] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7108.571625] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7108.577157] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7108.582864] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7108.590377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7108.593952] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7108.597740] Call Trace: [ 7108.602029] [ 7108.603517] ? show_regs.cold.9+0x22/0x2f [ 7108.606362] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7108.610456] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7108.612722] ? watchdog+0x40/0x40 [ 7108.614510] ? hrtimer_interrupt+0x128/0x2c0 [ 7108.620883] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7108.623565] ? apic_timer_interrupt+0xf/0x20 [ 7108.625698] [ 7108.626682] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7108.629332] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7108.631907] do_raw_spin_lock+0x177/0x190 [ 7108.634089] _raw_spin_lock+0x21/0x30 [ 7108.636168] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7108.639222] obd_mod_free+0x46/0x140 [ptlrpc] [ 7108.641729] mdc_commit_open+0x91/0xb0 [mdc] [ 7108.648658] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7108.653057] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7108.659889] after_reply+0xe2c/0x1780 [ptlrpc] [ 7108.662551] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7108.666651] ? __next_timer_interrupt+0x160/0x160 [ 7108.669373] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7108.671944] ? do_wait_intr+0xf0/0xf0 [ 7108.673958] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7108.681382] kthread+0x1d1/0x200 [ 7108.682351] ? set_kthread_struct+0x70/0x70 [ 7108.683815] ret_from_fork+0x1f/0x30 [ 7108.687176] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4595s! [ 7108.700337] Showing busy workqueues and worker pools: [ 7108.706270] workqueue events: flags=0x0 [ 7108.712600] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7108.721063] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7108.727188] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7108.733855] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7108.739234] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7108.742987] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7108.748135] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7108.751602] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7108.759584] workqueue events_long: flags=0x0 [ 7108.761985] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7108.767253] pending: lu_objects_flush [obdclass] [ 7108.770032] workqueue events_unbound: flags=0x2 [ 7108.772421] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7108.778382] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7108.781795] workqueue events_power_efficient: flags=0x80 [ 7108.784433] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7108.787507] pending: neigh_periodic_work, do_cache_clean [ 7108.789608] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7108.795535] pending: neigh_periodic_work, check_lifetime [ 7108.798614] workqueue mm_percpu_wq: flags=0x8 [ 7108.803584] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7108.807001] pending: vmstat_update, lru_add_drain_per_cpu [ 7108.810046] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7108.815493] pending: vmstat_update [ 7108.817907] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7108.821518] pending: lru_add_drain_per_cpu BAR(48) [ 7108.826476] workqueue ipv6_addrconf: flags=0x40008 [ 7108.830844] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7108.834350] pending: addrconf_verify_work [ 7108.836511] workqueue knbd0-recv: flags=0x1a [ 7108.838616] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7108.841353] in-flight: 650:recv_work [ 7108.842955] workqueue ptlrpc_pinger: flags=0x2 [ 7108.844856] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7108.849422] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7108.857775] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4595s workers=4 idle: 36 466 29757 [ 7108.860500] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7098s workers=2 idle: 652 [ 7128.063787] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7128.071561] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7128.099306] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7128.112092] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7128.118472] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7128.124923] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7128.133868] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7128.139339] RAX: 000000000000418e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7128.152201] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7128.156137] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7128.167079] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7128.172180] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7128.176122] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7128.179562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7128.182261] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7128.185450] Call Trace: [ 7128.186561] [ 7128.187831] ? show_regs.cold.9+0x22/0x2f [ 7128.189780] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7128.192184] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7128.194450] ? watchdog+0x40/0x40 [ 7128.196295] ? hrtimer_interrupt+0x128/0x2c0 [ 7128.198276] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7128.204482] ? apic_timer_interrupt+0xf/0x20 [ 7128.215113] [ 7128.218493] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7128.229451] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7128.233361] do_raw_spin_lock+0x177/0x190 [ 7128.239425] _raw_spin_lock+0x21/0x30 [ 7128.243520] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7128.248836] ? do_raw_spin_unlock+0x75/0x190 [ 7128.252477] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7128.256879] ? do_wait_intr+0xf0/0xf0 [ 7128.259527] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7128.263061] mdc_reint+0x2d/0x110 [mdc] [ 7128.266524] mdc_unlink+0x658/0x870 [mdc] [ 7128.270754] lmv_unlink+0x34d/0xbd0 [lmv] [ 7128.281009] ll_unlink+0x24d/0x7f0 [lustre] [ 7128.284557] vfs_unlink+0x202/0x330 [ 7128.290529] do_unlinkat+0x2fa/0x460 [ 7128.295702] __x64_sys_unlink+0x37/0x50 [ 7128.299680] do_syscall_64+0xc1/0x440 [ 7128.302418] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7128.310410] RIP: 0033:0x7f83e5f5c10b [ 7128.312207] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7128.321704] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7128.325275] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7128.329761] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7128.333220] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7128.340145] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7128.344398] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7136.056505] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 7136.060486] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7136.061798] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 7136.061848] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 7136.061854] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7136.062248] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7136.062260] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7136.062266] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7136.062581] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7136.062592] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7136.062596] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7136.062600] RAX: 000000000000510d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7136.062603] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7136.062605] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7136.062608] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7136.062610] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7136.062614] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7136.062617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7136.062620] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7136.062629] Call Trace: [ 7136.062635] [ 7136.062640] ? show_regs.cold.9+0x22/0x2f [ 7136.062654] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7136.062660] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7136.062668] ? watchdog+0x40/0x40 [ 7136.062673] ? hrtimer_interrupt+0x128/0x2c0 [ 7136.062678] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7136.062689] ? apic_timer_interrupt+0xf/0x20 [ 7136.062698] [ 7136.062701] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7136.062710] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7136.062717] do_raw_spin_lock+0x177/0x190 [ 7136.062721] _raw_spin_lock+0x21/0x30 [ 7136.062737] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7136.063854] process_one_work+0x2c8/0x700 [ 7136.066034] worker_thread+0x296/0x6e0 [ 7136.066040] ? rescuer_thread+0x570/0x570 [ 7136.066044] kthread+0x1d1/0x200 [ 7136.066049] ? set_kthread_struct+0x70/0x70 [ 7136.066052] ret_from_fork+0x1f/0x30 [ 7136.086802] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7136.086821] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7136.086824] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7136.086838] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7136.086842] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7136.086847] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7136.086850] RDX: 0000000000004f5b RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7136.086852] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7136.086854] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7136.086857] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7136.087016] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7136.087019] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7136.087021] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7136.087035] Call Trace: [ 7136.087041] [ 7136.087046] ? show_regs.cold.9+0x22/0x2f [ 7136.087056] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7136.087062] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7136.087067] ? watchdog+0x40/0x40 [ 7136.087071] ? hrtimer_interrupt+0x128/0x2c0 [ 7136.087075] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7136.087084] ? apic_timer_interrupt+0xf/0x20 [ 7136.087088] [ 7136.087091] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7136.087097] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7136.087102] do_raw_spin_lock+0x177/0x190 [ 7136.087107] _raw_spin_lock+0x21/0x30 [ 7136.087114] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7136.087417] obd_mod_free+0x46/0x140 [ptlrpc] [ 7136.087771] mdc_commit_open+0x91/0xb0 [mdc] [ 7136.087808] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7136.088093] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7136.093121] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7136.093201] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7136.093209] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7136.093213] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7136.093228] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7136.093232] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7136.093236] RAX: 00000000000045c1 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7136.093239] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7136.093241] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7136.093243] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7136.093245] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7136.093247] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7136.093250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7136.093252] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7136.093259] Call Trace: [ 7136.093264] [ 7136.093269] ? show_regs.cold.9+0x22/0x2f [ 7136.093277] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7136.093283] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7136.093287] ? watchdog+0x40/0x40 [ 7136.093291] ? hrtimer_interrupt+0x128/0x2c0 [ 7136.093294] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7136.093304] ? apic_timer_interrupt+0xf/0x20 [ 7136.093309] [ 7136.093312] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7136.093316] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7136.093322] do_raw_spin_lock+0x177/0x190 [ 7136.093326] _raw_spin_lock+0x21/0x30 [ 7136.093331] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7136.094072] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7136.094285] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7136.094510] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7136.094766] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7136.095067] ? woken_wake_function+0x30/0x30 [ 7136.095076] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7136.095295] kthread+0x1d1/0x200 [ 7136.095301] ? set_kthread_struct+0x70/0x70 [ 7136.095305] ret_from_fork+0x1f/0x30 [ 7136.544570] after_reply+0xe2c/0x1780 [ptlrpc] [ 7136.550307] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7136.557687] ? __next_timer_interrupt+0x160/0x160 [ 7136.561772] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7136.572587] ? do_wait_intr+0xf0/0xf0 [ 7136.574405] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7136.578480] kthread+0x1d1/0x200 [ 7136.580408] ? set_kthread_struct+0x70/0x70 [ 7136.581600] ret_from_fork+0x1f/0x30 [ 7139.299075] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4626s! [ 7139.302448] Showing busy workqueues and worker pools: [ 7139.307589] workqueue events: flags=0x0 [ 7139.310011] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7139.331662] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7139.337883] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7139.343066] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7139.347263] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7139.354795] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7139.359314] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7139.364979] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7139.370763] workqueue events_long: flags=0x0 [ 7139.376266] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7139.382435] pending: lu_objects_flush [obdclass] [ 7139.385161] workqueue events_unbound: flags=0x2 [ 7139.387732] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7139.395339] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7139.398743] workqueue events_power_efficient: flags=0x80 [ 7139.403866] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7139.412440] pending: neigh_periodic_work, do_cache_clean [ 7139.417175] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7139.422991] pending: neigh_periodic_work, check_lifetime [ 7139.425264] workqueue mm_percpu_wq: flags=0x8 [ 7139.427528] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7139.430848] pending: vmstat_update, lru_add_drain_per_cpu [ 7139.434966] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7139.439722] pending: vmstat_update [ 7139.442588] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7139.448139] pending: lru_add_drain_per_cpu BAR(48) [ 7139.451348] workqueue ipv6_addrconf: flags=0x40008 [ 7139.459878] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7139.466957] pending: addrconf_verify_work [ 7139.469833] workqueue knbd0-recv: flags=0x1a [ 7139.474212] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7139.479083] in-flight: 650:recv_work [ 7139.482032] workqueue ptlrpc_pinger: flags=0x2 [ 7139.486892] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7139.496533] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7139.498720] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4626s workers=4 idle: 36 466 29757 [ 7139.504197] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7129s workers=2 idle: 652 [ 7156.063168] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7156.069267] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7156.113716] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7156.123229] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7156.129047] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7156.133057] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7156.149804] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7156.154955] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7156.160549] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7156.165199] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7156.169376] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7156.174820] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7156.178810] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7156.187790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7156.192636] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7156.198334] Call Trace: [ 7156.199940] [ 7156.201696] ? show_regs.cold.9+0x22/0x2f [ 7156.205144] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7156.209123] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7156.211364] ? watchdog+0x40/0x40 [ 7156.213621] ? hrtimer_interrupt+0x128/0x2c0 [ 7156.215557] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7156.218715] ? apic_timer_interrupt+0xf/0x20 [ 7156.221263] [ 7156.223465] ? native_safe_halt+0x2e/0x50 [ 7156.225429] kvm_wait+0xcd/0xe0 [ 7156.227311] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7156.229438] do_raw_spin_lock+0x177/0x190 [ 7156.232570] _raw_spin_lock+0x21/0x30 [ 7156.234491] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7156.239210] ? do_raw_spin_unlock+0x75/0x190 [ 7156.242049] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7156.245174] ? do_wait_intr+0xf0/0xf0 [ 7156.246959] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7156.249635] mdc_reint+0x2d/0x110 [mdc] [ 7156.254175] mdc_unlink+0x658/0x870 [mdc] [ 7156.258939] lmv_unlink+0x34d/0xbd0 [lmv] [ 7156.261770] ll_unlink+0x24d/0x7f0 [lustre] [ 7156.264770] vfs_unlink+0x202/0x330 [ 7156.268316] do_unlinkat+0x2fa/0x460 [ 7156.274818] __x64_sys_unlink+0x37/0x50 [ 7156.278547] do_syscall_64+0xc1/0x440 [ 7156.283449] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7156.287595] RIP: 0033:0x7f83e5f5c10b [ 7156.289741] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7156.299793] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7156.303724] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7156.307498] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7156.310973] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7156.319196] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7156.325647] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7164.058628] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 7164.059038] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 7164.060060] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 7164.060069] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7164.060137] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7164.060143] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7164.060149] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7164.060430] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7164.060440] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7164.060443] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7164.060448] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7164.060450] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7164.060453] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7164.060472] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7164.060474] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7164.060478] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7164.060481] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7164.060483] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7164.060491] Call Trace: [ 7164.060496] [ 7164.060502] ? show_regs.cold.9+0x22/0x2f [ 7164.060511] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7164.060518] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7164.060523] ? watchdog+0x40/0x40 [ 7164.060528] ? hrtimer_interrupt+0x128/0x2c0 [ 7164.060532] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7164.060539] ? apic_timer_interrupt+0xf/0x20 [ 7164.060544] [ 7164.060548] ? native_safe_halt+0x2e/0x50 [ 7164.060553] kvm_wait+0xcd/0xe0 [ 7164.060560] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7164.060568] do_raw_spin_lock+0x177/0x190 [ 7164.060573] _raw_spin_lock+0x21/0x30 [ 7164.060577] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7164.060780] process_one_work+0x2c8/0x700 [ 7164.060788] worker_thread+0x296/0x6e0 [ 7164.060793] ? rescuer_thread+0x570/0x570 [ 7164.060797] kthread+0x1d1/0x200 [ 7164.060801] ? set_kthread_struct+0x70/0x70 [ 7164.060805] ret_from_fork+0x1f/0x30 [ 7164.066528] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7164.066608] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7164.066615] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7164.066618] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7164.066631] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7164.066635] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7164.066639] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7164.066642] RDX: 0000000000000ac0 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7164.066644] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7164.066647] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7164.066649] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7164.066652] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7164.075609] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7164.075914] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7164.075921] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7164.075924] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7164.075938] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7164.076081] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7164.076086] RAX: 0000000000007712 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7164.076088] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7164.076091] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7164.076093] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7164.076095] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7164.076098] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7164.076100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7164.076102] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7164.076110] Call Trace: [ 7164.076117] [ 7164.076123] ? show_regs.cold.9+0x22/0x2f [ 7164.076133] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7164.076140] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7164.076145] ? watchdog+0x40/0x40 [ 7164.076150] ? hrtimer_interrupt+0x128/0x2c0 [ 7164.076154] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7164.076161] ? apic_timer_interrupt+0xf/0x20 [ 7164.076165] [ 7164.076168] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7164.076172] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7164.076178] do_raw_spin_lock+0x177/0x190 [ 7164.076183] _raw_spin_lock+0x21/0x30 [ 7164.076188] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7164.076553] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7164.076879] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7164.084071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7164.084082] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7164.084090] Call Trace: [ 7164.084097] [ 7164.084104] ? show_regs.cold.9+0x22/0x2f [ 7164.084117] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7164.084124] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7164.084129] ? watchdog+0x40/0x40 [ 7164.084133] ? hrtimer_interrupt+0x128/0x2c0 [ 7164.084138] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7164.084149] ? apic_timer_interrupt+0xf/0x20 [ 7164.084155] [ 7164.084158] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7164.084164] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7164.084169] do_raw_spin_lock+0x177/0x190 [ 7164.084174] _raw_spin_lock+0x21/0x30 [ 7164.084181] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7164.084816] obd_mod_free+0x46/0x140 [ptlrpc] [ 7164.085817] mdc_commit_open+0x91/0xb0 [mdc] [ 7164.085915] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7164.086196] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7164.086402] after_reply+0xe2c/0x1780 [ptlrpc] [ 7164.115369] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7164.120399] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7164.125349] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7164.125885] ? woken_wake_function+0x30/0x30 [ 7164.125895] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7164.126239] kthread+0x1d1/0x200 [ 7164.126246] ? set_kthread_struct+0x70/0x70 [ 7164.126250] ret_from_fork+0x1f/0x30 [ 7164.536348] ? __next_timer_interrupt+0x160/0x160 [ 7164.546432] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7164.548597] ? do_wait_intr+0xf0/0xf0 [ 7164.556592] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7164.558372] kthread+0x1d1/0x200 [ 7164.559361] ? set_kthread_struct+0x70/0x70 [ 7164.565567] ret_from_fork+0x1f/0x30 [ 7170.021152] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4656s! [ 7170.027978] Showing busy workqueues and worker pools: [ 7170.030743] workqueue events: flags=0x0 [ 7170.034047] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7170.039878] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7170.047979] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7170.056537] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7170.063441] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7170.067550] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7170.074147] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7170.079979] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7170.085460] workqueue events_long: flags=0x0 [ 7170.090043] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7170.095095] pending: lu_objects_flush [obdclass] [ 7170.099223] workqueue events_unbound: flags=0x2 [ 7170.103427] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7170.108794] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7170.113913] workqueue events_power_efficient: flags=0x80 [ 7170.118535] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7170.124709] pending: neigh_periodic_work, do_cache_clean [ 7170.129228] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7170.133765] pending: neigh_periodic_work, check_lifetime [ 7170.138132] workqueue mm_percpu_wq: flags=0x8 [ 7170.141735] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7170.147026] pending: vmstat_update, lru_add_drain_per_cpu [ 7170.150958] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7170.154271] pending: vmstat_update [ 7170.156708] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7170.169423] pending: lru_add_drain_per_cpu BAR(48) [ 7170.176161] workqueue ipv6_addrconf: flags=0x40008 [ 7170.183525] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7170.193836] pending: addrconf_verify_work [ 7170.201055] workqueue knbd0-recv: flags=0x1a [ 7170.207475] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7170.212210] in-flight: 650:recv_work [ 7170.219833] workqueue ptlrpc_pinger: flags=0x2 [ 7170.223732] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7170.232050] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7170.247550] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4657s workers=4 idle: 36 466 29757 [ 7170.262091] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7160s workers=2 idle: 652 [ 7184.063225] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7184.066777] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7184.105734] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7184.118397] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7184.132751] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7184.148167] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7184.180134] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7184.182429] RAX: 0000000000007d0e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7184.190651] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7184.195870] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7184.210819] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7184.217794] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7184.225544] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7184.233754] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7184.245172] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7184.252005] Call Trace: [ 7184.255997] [ 7184.257567] ? show_regs.cold.9+0x22/0x2f [ 7184.260430] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7184.267908] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7184.270739] ? watchdog+0x40/0x40 [ 7184.272972] ? hrtimer_interrupt+0x128/0x2c0 [ 7184.276952] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7184.280851] ? apic_timer_interrupt+0xf/0x20 [ 7184.281924] [ 7184.289964] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7184.292602] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7184.299982] do_raw_spin_lock+0x177/0x190 [ 7184.305686] _raw_spin_lock+0x21/0x30 [ 7184.309337] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7184.316828] ? do_raw_spin_unlock+0x75/0x190 [ 7184.321982] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7184.330033] ? do_wait_intr+0xf0/0xf0 [ 7184.337366] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7184.342483] mdc_reint+0x2d/0x110 [mdc] [ 7184.345847] mdc_unlink+0x658/0x870 [mdc] [ 7184.353068] lmv_unlink+0x34d/0xbd0 [lmv] [ 7184.356076] ll_unlink+0x24d/0x7f0 [lustre] [ 7184.359840] vfs_unlink+0x202/0x330 [ 7184.361095] do_unlinkat+0x2fa/0x460 [ 7184.368374] __x64_sys_unlink+0x37/0x50 [ 7184.371198] do_syscall_64+0xc1/0x440 [ 7184.375107] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7184.385891] RIP: 0033:0x7f83e5f5c10b [ 7184.388832] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7184.404947] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7184.416530] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7184.419231] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7184.436851] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7184.443028] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7184.456039] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7192.060944] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 7192.064304] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 7192.071937] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7192.072032] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7192.072040] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7192.072048] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7192.078669] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7192.078752] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7192.078761] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7192.078764] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7192.078780] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7192.078784] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7192.078789] RAX: 0000000000006e7c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7192.078791] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7192.078794] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7192.078796] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7192.078799] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7192.078803] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7192.078806] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7192.078808] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7192.078818] Call Trace: [ 7192.078824] [ 7192.078830] ? show_regs.cold.9+0x22/0x2f [ 7192.078841] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7192.078847] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7192.078852] ? watchdog+0x40/0x40 [ 7192.078856] ? hrtimer_interrupt+0x128/0x2c0 [ 7192.078872] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7192.078880] ? apic_timer_interrupt+0xf/0x20 [ 7192.078885] [ 7192.078888] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7192.078894] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7192.078900] do_raw_spin_lock+0x177/0x190 [ 7192.078906] _raw_spin_lock+0x21/0x30 [ 7192.078911] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7192.079992] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7192.080499] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7192.080702] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7192.080949] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7192.081539] ? woken_wake_function+0x30/0x30 [ 7192.081548] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7192.081758] kthread+0x1d1/0x200 [ 7192.081763] ? set_kthread_struct+0x70/0x70 [ 7192.081767] ret_from_fork+0x1f/0x30 [ 7192.290663] [ 7192.291678] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7192.295121] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7192.306227] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7192.310686] RAX: 0000000000002498 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7192.316207] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7192.321194] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7192.324620] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7192.328322] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7192.331948] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7192.335791] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7192.340776] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7192.346422] Call Trace: [ 7192.348634] [ 7192.349983] ? show_regs.cold.9+0x22/0x2f [ 7192.353360] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7192.357053] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7192.359393] ? watchdog+0x40/0x40 [ 7192.362343] ? hrtimer_interrupt+0x128/0x2c0 [ 7192.364697] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7192.366448] ? apic_timer_interrupt+0xf/0x20 [ 7192.368844] [ 7192.369903] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7192.372743] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7192.375399] do_raw_spin_lock+0x177/0x190 [ 7192.377259] _raw_spin_lock+0x21/0x30 [ 7192.379242] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7192.382148] process_one_work+0x2c8/0x700 [ 7192.384217] worker_thread+0x296/0x6e0 [ 7192.385855] ? rescuer_thread+0x570/0x570 [ 7192.387789] kthread+0x1d1/0x200 [ 7192.389286] ? set_kthread_struct+0x70/0x70 [ 7192.391602] ret_from_fork+0x1f/0x30 [ 7196.057069] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 7196.075835] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7196.122682] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7196.131109] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7196.141463] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7196.146342] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7196.170646] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7196.180347] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7196.192656] RDX: 0000000000006eb5 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7196.201568] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7196.214743] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7196.225059] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7196.234771] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7196.247281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7196.253176] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7196.259320] Call Trace: [ 7196.261460] [ 7196.264885] ? show_regs.cold.9+0x22/0x2f [ 7196.268868] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7196.273586] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7196.281705] ? watchdog+0x40/0x40 [ 7196.283111] ? hrtimer_interrupt+0x128/0x2c0 [ 7196.293325] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7196.303003] ? apic_timer_interrupt+0xf/0x20 [ 7196.307443] [ 7196.308212] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7196.318424] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7196.325027] do_raw_spin_lock+0x177/0x190 [ 7196.331940] _raw_spin_lock+0x21/0x30 [ 7196.340210] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7196.345302] obd_mod_free+0x46/0x140 [ptlrpc] [ 7196.353888] mdc_commit_open+0x91/0xb0 [mdc] [ 7196.361054] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7196.369913] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7196.380643] after_reply+0xe2c/0x1780 [ptlrpc] [ 7196.386063] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7196.397596] ? __next_timer_interrupt+0x160/0x160 [ 7196.407639] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7196.414806] ? do_wait_intr+0xf0/0xf0 [ 7196.418528] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7196.427931] kthread+0x1d1/0x200 [ 7196.436329] ? set_kthread_struct+0x70/0x70 [ 7196.443065] ret_from_fork+0x1f/0x30 [ 7200.736428] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4687s! [ 7200.748859] Showing busy workqueues and worker pools: [ 7200.752881] workqueue events: flags=0x0 [ 7200.757186] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7200.762149] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7200.769877] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7200.775857] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7200.782363] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7200.787425] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7200.797153] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7200.800034] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7200.811819] workqueue events_long: flags=0x0 [ 7200.817194] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7200.828877] pending: lu_objects_flush [obdclass] [ 7200.834013] workqueue events_unbound: flags=0x2 [ 7200.839778] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7200.846236] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7200.855136] workqueue events_power_efficient: flags=0x80 [ 7200.859486] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7200.862227] pending: neigh_periodic_work, do_cache_clean [ 7200.866059] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7200.869717] pending: neigh_periodic_work, check_lifetime [ 7200.873349] workqueue mm_percpu_wq: flags=0x8 [ 7200.879174] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7200.891977] pending: vmstat_update, lru_add_drain_per_cpu [ 7200.896189] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7200.900825] pending: vmstat_update [ 7200.903124] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7200.909195] pending: lru_add_drain_per_cpu BAR(48) [ 7200.913780] workqueue ipv6_addrconf: flags=0x40008 [ 7200.919447] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7200.924712] pending: addrconf_verify_work [ 7200.928668] workqueue knbd0-recv: flags=0x1a [ 7200.932143] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7200.934967] in-flight: 650:recv_work [ 7200.937483] workqueue ptlrpc_pinger: flags=0x2 [ 7200.940248] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7200.946439] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7200.951393] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4687s workers=4 idle: 36 466 29757 [ 7200.954243] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7191s workers=2 idle: 652 [ 7212.062416] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7212.069834] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7212.132763] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7212.140879] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7212.153975] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7212.159183] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7212.173086] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7212.180386] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7212.184776] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7212.188676] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7212.192619] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7212.196364] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7212.199248] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7212.208892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7212.214884] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7212.219336] Call Trace: [ 7212.221647] [ 7212.222549] ? show_regs.cold.9+0x22/0x2f [ 7212.224288] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7212.229573] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7212.234297] ? watchdog+0x40/0x40 [ 7212.235311] ? hrtimer_interrupt+0x128/0x2c0 [ 7212.236486] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7212.237863] ? apic_timer_interrupt+0xf/0x20 [ 7212.239061] [ 7212.239653] ? native_safe_halt+0x2e/0x50 [ 7212.240954] kvm_wait+0xcd/0xe0 [ 7212.242597] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7212.245064] do_raw_spin_lock+0x177/0x190 [ 7212.246861] _raw_spin_lock+0x21/0x30 [ 7212.248298] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7212.250464] ? do_raw_spin_unlock+0x75/0x190 [ 7212.252197] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7212.254794] ? do_wait_intr+0xf0/0xf0 [ 7212.256468] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7212.258852] mdc_reint+0x2d/0x110 [mdc] [ 7212.260683] mdc_unlink+0x658/0x870 [mdc] [ 7212.264644] lmv_unlink+0x34d/0xbd0 [lmv] [ 7212.271031] ll_unlink+0x24d/0x7f0 [lustre] [ 7212.274922] vfs_unlink+0x202/0x330 [ 7212.277165] do_unlinkat+0x2fa/0x460 [ 7212.280785] __x64_sys_unlink+0x37/0x50 [ 7212.282836] do_syscall_64+0xc1/0x440 [ 7212.284487] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7212.286494] RIP: 0033:0x7f83e5f5c10b [ 7212.288291] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7212.300349] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7212.303775] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7212.306800] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7212.308742] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7212.314109] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7212.319678] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7220.060169] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 7220.060448] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 7220.070731] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7220.070828] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7220.074838] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7220.075048] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7220.075055] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7220.075058] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7220.075072] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7220.075076] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 [ 7220.117443] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7220.117458] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7220.117844] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7220.117857] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7220.117861] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7220.117865] RAX: 0000000000001484 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7220.117868] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7220.117871] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7220.117873] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7220.117876] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7220.117879] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7220.117882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7220.117884] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7220.117893] Call Trace: [ 7220.117898] [ 7220.117903] ? show_regs.cold.9+0x22/0x2f [ 7220.117923] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7220.117930] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7220.117934] ? watchdog+0x40/0x40 [ 7220.117938] ? hrtimer_interrupt+0x128/0x2c0 [ 7220.117942] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7220.118023] ? apic_timer_interrupt+0xf/0x20 [ 7220.118030] [ 7220.118033] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7220.118037] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7220.118043] do_raw_spin_lock+0x177/0x190 [ 7220.118048] _raw_spin_lock+0x21/0x30 [ 7220.118056] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7220.118479] process_one_work+0x2c8/0x700 [ 7220.118487] worker_thread+0x296/0x6e0 [ 7220.118492] ? rescuer_thread+0x570/0x570 [ 7220.118495] kthread+0x1d1/0x200 [ 7220.118504] ? set_kthread_struct+0x70/0x70 [ 7220.118508] ret_from_fork+0x1f/0x30 [ 7220.273894] ORIG_RAX: ffffffffffffff13 [ 7220.277813] RAX: 0000000000007d87 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7220.280817] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7220.283952] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7220.287599] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7220.292451] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7220.295446] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7220.298388] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7220.300681] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7220.305731] Call Trace: [ 7220.307820] [ 7220.308706] ? show_regs.cold.9+0x22/0x2f [ 7220.312146] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7220.313744] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7220.316050] ? watchdog+0x40/0x40 [ 7220.317318] ? hrtimer_interrupt+0x128/0x2c0 [ 7220.318733] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7220.321866] ? apic_timer_interrupt+0xf/0x20 [ 7220.323681] [ 7220.324756] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7220.327059] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7220.329192] do_raw_spin_lock+0x177/0x190 [ 7220.330625] _raw_spin_lock+0x21/0x30 [ 7220.331884] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7220.334618] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7220.337304] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7220.339812] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7220.342568] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7220.345045] ? woken_wake_function+0x30/0x30 [ 7220.346782] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7220.349698] kthread+0x1d1/0x200 [ 7220.351069] ? set_kthread_struct+0x70/0x70 [ 7220.352860] ret_from_fork+0x1f/0x30 [ 7228.056256] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 7228.059445] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7228.079726] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7228.083937] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7228.087010] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7228.089268] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7228.095859] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7228.099789] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7228.101666] RDX: 000000000000612e RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7228.103863] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7228.106522] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7228.108841] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7228.111211] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7228.113830] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7228.115377] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7228.117858] Call Trace: [ 7228.118598] [ 7228.119280] ? show_regs.cold.9+0x22/0x2f [ 7228.120369] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7228.121805] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7228.123358] ? watchdog+0x40/0x40 [ 7228.124441] ? hrtimer_interrupt+0x128/0x2c0 [ 7228.125637] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7228.127294] ? apic_timer_interrupt+0xf/0x20 [ 7228.129223] [ 7228.130307] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7228.132742] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7228.134617] do_raw_spin_lock+0x177/0x190 [ 7228.136332] _raw_spin_lock+0x21/0x30 [ 7228.137467] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7228.139384] obd_mod_free+0x46/0x140 [ptlrpc] [ 7228.141376] mdc_commit_open+0x91/0xb0 [mdc] [ 7228.142942] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7228.144826] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7228.147044] after_reply+0xe2c/0x1780 [ptlrpc] [ 7228.148650] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7228.150546] ? __next_timer_interrupt+0x160/0x160 [ 7228.152440] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7228.154207] ? do_wait_intr+0xf0/0xf0 [ 7228.155289] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7228.157406] kthread+0x1d1/0x200 [ 7228.158467] ? set_kthread_struct+0x70/0x70 [ 7228.159994] ret_from_fork+0x1f/0x30 [ 7231.456098] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4718s! [ 7231.458373] Showing busy workqueues and worker pools: [ 7231.459985] workqueue events: flags=0x0 [ 7231.461194] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7231.463793] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7231.466839] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7231.469281] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7231.471608] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7231.473685] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7231.477066] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7231.479209] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7231.482771] workqueue events_long: flags=0x0 [ 7231.484128] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7231.486283] pending: lu_objects_flush [obdclass] [ 7231.488155] workqueue events_unbound: flags=0x2 [ 7231.489796] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7231.492124] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7231.494012] workqueue events_power_efficient: flags=0x80 [ 7231.495653] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7231.498061] pending: neigh_periodic_work, do_cache_clean [ 7231.500305] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7231.503051] pending: neigh_periodic_work, check_lifetime [ 7231.505060] workqueue mm_percpu_wq: flags=0x8 [ 7231.513882] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7231.519631] pending: vmstat_update, lru_add_drain_per_cpu [ 7231.521994] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7231.524234] pending: vmstat_update [ 7231.526065] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7231.528494] pending: lru_add_drain_per_cpu BAR(48) [ 7231.530552] workqueue ipv6_addrconf: flags=0x40008 [ 7231.533246] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7231.536367] pending: addrconf_verify_work [ 7231.539900] workqueue knbd0-recv: flags=0x1a [ 7231.543590] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7231.550445] in-flight: 650:recv_work [ 7231.555964] workqueue ptlrpc_pinger: flags=0x2 [ 7231.557864] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7231.563301] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7231.568624] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4718s workers=4 idle: 36 466 29757 [ 7231.577740] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7221s workers=2 idle: 652 [ 7240.065912] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7240.072374] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7240.092641] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7240.096805] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7240.099983] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7240.102179] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7240.109687] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7240.112332] RAX: 0000000000001da7 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7240.114784] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7240.117222] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7240.119586] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7240.121769] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7240.123850] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7240.126546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7240.128755] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7240.131155] Call Trace: [ 7240.132095] [ 7240.132829] ? show_regs.cold.9+0x22/0x2f [ 7240.134267] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7240.135808] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7240.137475] ? watchdog+0x40/0x40 [ 7240.138709] ? hrtimer_interrupt+0x128/0x2c0 [ 7240.140265] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7240.141973] ? apic_timer_interrupt+0xf/0x20 [ 7240.143446] [ 7240.144251] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7240.146183] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7240.148048] do_raw_spin_lock+0x177/0x190 [ 7240.149385] _raw_spin_lock+0x21/0x30 [ 7240.150791] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7240.152734] ? do_raw_spin_unlock+0x75/0x190 [ 7240.154375] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7240.156126] ? do_wait_intr+0xf0/0xf0 [ 7240.157768] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7240.159507] mdc_reint+0x2d/0x110 [mdc] [ 7240.160695] mdc_unlink+0x658/0x870 [mdc] [ 7240.161997] lmv_unlink+0x34d/0xbd0 [lmv] [ 7240.163226] ll_unlink+0x24d/0x7f0 [lustre] [ 7240.164765] vfs_unlink+0x202/0x330 [ 7240.166034] do_unlinkat+0x2fa/0x460 [ 7240.167381] __x64_sys_unlink+0x37/0x50 [ 7240.168903] do_syscall_64+0xc1/0x440 [ 7240.170342] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7240.172115] RIP: 0033:0x7f83e5f5c10b [ 7240.173480] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7240.180701] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7240.183336] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7240.185642] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7240.188389] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7240.190564] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7240.192708] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7248.059012] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 7248.062885] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 7248.063182] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7248.063254] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7248.063260] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7248.063263] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7248.063278] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7248.063282] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7248.063287] RAX: 000000000000460f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7248.063290] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7248.063292] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7248.063294] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7248.063297] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7248.063300] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7248.063303] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7248.063305] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7248.063313] Call Trace: [ 7248.063319] [ 7248.063325] ? show_regs.cold.9+0x22/0x2f [ 7248.063335] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7248.063341] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7248.063345] ? watchdog+0x40/0x40 [ 7248.063349] ? hrtimer_interrupt+0x128/0x2c0 [ 7248.063353] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7248.063362] ? apic_timer_interrupt+0xf/0x20 [ 7248.063367] [ 7248.063494] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7248.063500] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7248.063516] do_raw_spin_lock+0x177/0x190 [ 7248.063522] _raw_spin_lock+0x21/0x30 [ 7248.063526] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7248.063920] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7248.064233] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7248.064454] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7248.064797] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7248.065108] ? woken_wake_function+0x30/0x30 [ 7248.065241] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7248.065474] kthread+0x1d1/0x200 [ 7248.065481] ? set_kthread_struct+0x70/0x70 [ 7248.065484] ret_from_fork+0x1f/0x30 [ 7248.209150] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7248.229259] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7248.235088] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7248.239096] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7248.241837] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7248.243634] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7248.252069] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7248.256067] RAX: 0000000000003b0b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7248.259080] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7248.265791] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7248.278736] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7248.282320] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7248.284113] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7248.288450] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7248.290723] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7248.295145] Call Trace: [ 7248.296363] [ 7248.297095] ? show_regs.cold.9+0x22/0x2f [ 7248.298526] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7248.301033] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7248.305822] ? watchdog+0x40/0x40 [ 7248.307835] ? hrtimer_interrupt+0x128/0x2c0 [ 7248.309578] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7248.311510] ? apic_timer_interrupt+0xf/0x20 [ 7248.314865] [ 7248.315800] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7248.317603] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7248.321041] do_raw_spin_lock+0x177/0x190 [ 7248.326714] _raw_spin_lock+0x21/0x30 [ 7248.328608] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7248.331863] process_one_work+0x2c8/0x700 [ 7248.333822] worker_thread+0x296/0x6e0 [ 7248.335083] ? rescuer_thread+0x570/0x570 [ 7248.338272] kthread+0x1d1/0x200 [ 7248.339282] ? set_kthread_struct+0x70/0x70 [ 7248.341490] ret_from_fork+0x1f/0x30 [ 7256.056045] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 7256.060037] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7256.083886] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7256.089354] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7256.093886] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7256.097689] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7256.106246] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7256.109466] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7256.112531] RDX: 0000000000002ee2 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7256.115369] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7256.117663] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7256.121377] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7256.125042] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7256.128455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7256.131141] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7256.134550] Call Trace: [ 7256.135597] [ 7256.136501] ? show_regs.cold.9+0x22/0x2f [ 7256.138055] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7256.140265] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7256.142037] ? watchdog+0x40/0x40 [ 7256.143688] ? hrtimer_interrupt+0x128/0x2c0 [ 7256.145826] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7256.147901] ? apic_timer_interrupt+0xf/0x20 [ 7256.149596] [ 7256.150192] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7256.151949] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7256.154185] do_raw_spin_lock+0x177/0x190 [ 7256.155911] _raw_spin_lock+0x21/0x30 [ 7256.157213] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7256.159685] obd_mod_free+0x46/0x140 [ptlrpc] [ 7256.161705] mdc_commit_open+0x91/0xb0 [mdc] [ 7256.163559] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7256.165825] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7256.168508] after_reply+0xe2c/0x1780 [ptlrpc] [ 7256.170625] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7256.172669] ? __next_timer_interrupt+0x160/0x160 [ 7256.174689] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7256.176889] ? do_wait_intr+0xf0/0xf0 [ 7256.178960] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7256.181475] kthread+0x1d1/0x200 [ 7256.182894] ? set_kthread_struct+0x70/0x70 [ 7256.184646] ret_from_fork+0x1f/0x30 [ 7261.080042] rcu: INFO: rcu_sched self-detected stall on CPU [ 7261.085219] rcu: 0-....: (3358127 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 7261.093470] (t=5280127 jiffies g=262441 q=58691) [ 7261.103858] rcu: rcu_sched kthread starved for 4749324 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 7261.117531] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7261.124206] rcu: RCU grace-period kthread stack dump: [ 7261.137482] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7261.146957] Call Trace: [ 7261.147861] __schedule+0x351/0xcb0 [ 7261.157486] schedule+0xc0/0x180 [ 7261.163497] schedule_timeout+0xb4/0x190 [ 7261.167975] ? do_raw_spin_unlock+0x75/0x190 [ 7261.169928] ? __next_timer_interrupt+0x160/0x160 [ 7261.176470] ? prepare_to_swait_event+0xf3/0x140 [ 7261.178890] rcu_gp_kthread+0x8dd/0x15b0 [ 7261.187424] ? force_qs_rnp+0x2c0/0x2c0 [ 7261.190343] kthread+0x1d1/0x200 [ 7261.197944] ? set_kthread_struct+0x70/0x70 [ 7261.199885] ret_from_fork+0x1f/0x30 [ 7261.204923] rcu: Stack dump where RCU GP kthread last ran: [ 7261.207263] Sending NMI from CPU 0 to CPUs 2: [ 7261.210957] NMI backtrace for cpu 2 [ 7261.210964] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7261.210966] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7261.210968] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7261.210971] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7261.210972] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7261.210975] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 [ 7261.210977] RAX: 0000000000005f83 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7261.210978] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7261.210979] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7261.210981] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7261.210982] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7261.210984] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7261.210985] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7261.210986] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7261.210988] Call Trace: [ 7261.210989] [ 7261.210990] ? show_regs.cold.9+0x22/0x2f [ 7261.210991] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7261.210992] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7261.210993] ? nmi_handle+0xbd/0x210 [ 7261.210995] ? default_do_nmi+0x4b/0x1b0 [ 7261.210996] ? do_nmi+0x280/0x380 [ 7261.210996] ? end_repeat_nmi+0x16/0x1f [ 7261.210998] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7261.210999] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7261.210999] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7261.211000] [ 7261.211001] do_raw_spin_lock+0x177/0x190 [ 7261.211002] _raw_spin_lock+0x21/0x30 [ 7261.211003] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7261.211004] process_one_work+0x2c8/0x700 [ 7261.211005] worker_thread+0x296/0x6e0 [ 7261.211006] ? rescuer_thread+0x570/0x570 [ 7261.211007] kthread+0x1d1/0x200 [ 7261.211007] ? set_kthread_struct+0x70/0x70 [ 7261.211009] ret_from_fork+0x1f/0x30 [ 7261.217462] NMI backtrace for cpu 0 [ 7261.373563] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7261.384153] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7261.388052] Call Trace: [ 7261.395446] [ 7261.402997] ? dump_stack+0xbb/0x10e [ 7261.404212] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7261.407190] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7261.411055] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7261.418085] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7261.422382] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7261.424463] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7261.426455] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7261.428619] ? update_process_times+0x6d/0xc0 [ 7261.430456] ? tick_sched_handle.isra.18+0x27/0x90 [ 7261.438067] ? tick_sched_timer+0x53/0xb0 [ 7261.444441] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7261.446925] ? tick_sched_handle.isra.18+0x90/0x90 [ 7261.450332] ? hrtimer_interrupt+0x128/0x2c0 [ 7261.452081] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7261.456053] ? apic_timer_interrupt+0xf/0x20 [ 7261.459269] [ 7261.461591] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7261.471124] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7261.481051] ? do_raw_spin_lock+0x177/0x190 [ 7261.487359] ? _raw_spin_lock+0x21/0x30 [ 7261.491209] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7261.496163] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7261.499660] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7261.503648] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7261.506607] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7261.509614] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7261.512841] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7261.520500] ? __next_timer_interrupt+0x160/0x160 [ 7261.527640] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7261.540936] ? do_wait_intr+0xf0/0xf0 [ 7261.544903] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7261.550730] ? kthread+0x1d1/0x200 [ 7261.552176] ? set_kthread_struct+0x70/0x70 [ 7261.554147] ? ret_from_fork+0x1f/0x30 [ 7261.667625] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4748s! [ 7261.672353] Showing busy workqueues and worker pools: [ 7261.674986] workqueue events: flags=0x0 [ 7261.677272] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7261.681492] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7261.686751] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7261.690747] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7261.694292] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7261.698077] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7261.704746] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7261.712637] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7261.719034] workqueue events_long: flags=0x0 [ 7261.721168] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7261.726048] pending: lu_objects_flush [obdclass] [ 7261.729745] workqueue events_unbound: flags=0x2 [ 7261.732682] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7261.736795] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7261.739258] workqueue events_power_efficient: flags=0x80 [ 7261.742231] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7261.745451] pending: neigh_periodic_work, do_cache_clean [ 7261.748583] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7261.752340] pending: neigh_periodic_work, check_lifetime [ 7261.756219] workqueue mm_percpu_wq: flags=0x8 [ 7261.758770] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7261.762671] pending: vmstat_update, lru_add_drain_per_cpu [ 7261.765982] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7261.770657] pending: vmstat_update [ 7261.777301] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7261.783832] pending: lru_add_drain_per_cpu BAR(48) [ 7261.790261] workqueue ipv6_addrconf: flags=0x40008 [ 7261.797391] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7261.805208] pending: addrconf_verify_work [ 7261.809673] workqueue knbd0-recv: flags=0x1a [ 7261.815250] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7261.825016] in-flight: 650:recv_work [ 7261.831341] workqueue ptlrpc_pinger: flags=0x2 [ 7261.837020] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7261.847779] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7261.856319] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4748s workers=4 idle: 36 466 29757 [ 7261.861585] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7251s workers=2 idle: 652 [ 7268.062010] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7268.065520] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7268.086159] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7268.091294] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7268.095003] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7268.097928] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7268.105933] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7268.108631] RAX: 0000000000001083 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7268.111421] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7268.114825] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7268.118187] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7268.121596] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7268.125176] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7268.129054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7268.131594] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7268.134804] Call Trace: [ 7268.135770] [ 7268.136829] ? show_regs.cold.9+0x22/0x2f [ 7268.138891] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7268.140996] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7268.142962] ? watchdog+0x40/0x40 [ 7268.144531] ? hrtimer_interrupt+0x128/0x2c0 [ 7268.148017] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7268.150530] ? apic_timer_interrupt+0xf/0x20 [ 7268.152510] [ 7268.153618] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7268.156438] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7268.158892] do_raw_spin_lock+0x177/0x190 [ 7268.160796] _raw_spin_lock+0x21/0x30 [ 7268.162521] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7268.165108] ? do_raw_spin_unlock+0x75/0x190 [ 7268.167385] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7268.169838] ? do_wait_intr+0xf0/0xf0 [ 7268.171460] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7268.173879] mdc_reint+0x2d/0x110 [mdc] [ 7268.175829] mdc_unlink+0x658/0x870 [mdc] [ 7268.177813] lmv_unlink+0x34d/0xbd0 [lmv] [ 7268.180452] ll_unlink+0x24d/0x7f0 [lustre] [ 7268.182616] vfs_unlink+0x202/0x330 [ 7268.184715] do_unlinkat+0x2fa/0x460 [ 7268.186305] __x64_sys_unlink+0x37/0x50 [ 7268.188851] do_syscall_64+0xc1/0x440 [ 7268.190583] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7268.192978] RIP: 0033:0x7f83e5f5c10b [ 7268.194857] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7268.204797] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7268.208553] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7268.211924] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7268.215237] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7268.218505] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7268.221776] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7276.061309] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 7276.064827] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7276.090771] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7276.096760] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7276.100604] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7276.102902] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7276.111506] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7276.114576] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7276.118020] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7276.121774] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7276.125600] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7276.129403] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7276.132927] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7276.137329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7276.140627] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7276.144119] Call Trace: [ 7276.145172] [ 7276.147176] ? show_regs.cold.9+0x22/0x2f [ 7276.149165] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7276.150977] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7276.155166] ? watchdog+0x40/0x40 [ 7276.156824] ? hrtimer_interrupt+0x128/0x2c0 [ 7276.158995] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7276.163936] ? apic_timer_interrupt+0xf/0x20 [ 7276.169441] [ 7276.171845] ? native_safe_halt+0x2e/0x50 [ 7276.174876] kvm_wait+0xcd/0xe0 [ 7276.176060] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7276.177814] do_raw_spin_lock+0x177/0x190 [ 7276.181527] _raw_spin_lock+0x21/0x30 [ 7276.184096] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7276.186968] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7276.190497] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7276.193878] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7276.200171] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7276.202958] ? woken_wake_function+0x30/0x30 [ 7276.205655] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7276.210898] kthread+0x1d1/0x200 [ 7276.212488] ? set_kthread_struct+0x70/0x70 [ 7276.215384] ret_from_fork+0x1f/0x30 [ 7288.056099] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 7288.059290] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7288.060033] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 7288.081957] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7288.081973] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7288.081976] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7288.081991] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7288.081995] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7288.086406] Modules linked in: lustre(O) [ 7288.093289] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7288.093302] RDX: 0000000000000a79 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7288.093305] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7288.093308] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7288.093310] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7288.093314] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7288.093318] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7288.093321] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7288.093331] Call Trace: [ 7288.093339] [ 7288.093345] ? show_regs.cold.9+0x22/0x2f [ 7288.093358] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7288.093365] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7288.093370] ? watchdog+0x40/0x40 [ 7288.093374] ? hrtimer_interrupt+0x128/0x2c0 [ 7288.093384] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7288.093396] ? apic_timer_interrupt+0xf/0x20 [ 7288.093400] [ 7288.093404] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7288.093411] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7288.093415] do_raw_spin_lock+0x177/0x190 [ 7288.093420] _raw_spin_lock+0x21/0x30 [ 7288.093427] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7288.093931] obd_mod_free+0x46/0x140 [ptlrpc] [ 7288.094360] mdc_commit_open+0x91/0xb0 [mdc] [ 7288.094401] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7288.094755] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7288.095041] after_reply+0xe2c/0x1780 [ptlrpc] [ 7288.095626] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7288.095842] ? __next_timer_interrupt+0x160/0x160 [ 7288.095855] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7288.096391] ? do_wait_intr+0xf0/0xf0 [ 7288.096401] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7288.096742] kthread+0x1d1/0x200 [ 7288.096749] ? set_kthread_struct+0x70/0x70 [ 7288.096753] ret_from_fork+0x1f/0x30 [ 7288.209020] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7288.227626] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7288.232603] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7288.236064] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7288.238845] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7288.240616] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7288.247694] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7288.250585] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7288.253399] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7288.255712] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7288.258066] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7288.260626] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7288.263032] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7288.265939] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7288.268016] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7288.270734] Call Trace: [ 7288.271756] [ 7288.272441] ? show_regs.cold.9+0x22/0x2f [ 7288.274171] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7288.275881] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7288.277799] ? watchdog+0x40/0x40 [ 7288.279654] ? hrtimer_interrupt+0x128/0x2c0 [ 7288.281703] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7288.283735] ? apic_timer_interrupt+0xf/0x20 [ 7288.285592] [ 7288.286209] ? native_safe_halt+0x2e/0x50 [ 7288.287933] kvm_wait+0xcd/0xe0 [ 7288.289896] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7288.292620] do_raw_spin_lock+0x177/0x190 [ 7288.294349] _raw_spin_lock+0x21/0x30 [ 7288.295805] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7288.298718] process_one_work+0x2c8/0x700 [ 7288.300790] worker_thread+0x296/0x6e0 [ 7288.302997] ? rescuer_thread+0x570/0x570 [ 7288.305290] kthread+0x1d1/0x200 [ 7288.306313] ? set_kthread_struct+0x70/0x70 [ 7288.307899] ret_from_fork+0x1f/0x30 [ 7291.876092] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4778s! [ 7291.881781] Showing busy workqueues and worker pools: [ 7291.894401] workqueue events: flags=0x0 [ 7291.899181] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7291.908807] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7291.920233] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7291.925115] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7291.932075] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7291.938787] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7291.950375] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7291.955829] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7291.960947] workqueue events_long: flags=0x0 [ 7291.968281] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7291.980196] pending: lu_objects_flush [obdclass] [ 7291.983173] workqueue events_unbound: flags=0x2 [ 7291.989879] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7291.995909] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7291.998349] workqueue events_power_efficient: flags=0x80 [ 7292.000513] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7292.003851] pending: neigh_periodic_work, do_cache_clean [ 7292.009719] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7292.017249] pending: neigh_periodic_work, check_lifetime [ 7292.024197] workqueue mm_percpu_wq: flags=0x8 [ 7292.029746] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7292.037665] pending: vmstat_update, lru_add_drain_per_cpu [ 7292.042166] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7292.053524] pending: vmstat_update [ 7292.062716] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7292.066905] pending: lru_add_drain_per_cpu BAR(48) [ 7292.072071] workqueue ipv6_addrconf: flags=0x40008 [ 7292.077380] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7292.080231] pending: addrconf_verify_work [ 7292.082091] workqueue knbd0-recv: flags=0x1a [ 7292.085759] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7292.092343] in-flight: 650:recv_work [ 7292.098061] workqueue ptlrpc_pinger: flags=0x2 [ 7292.105657] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7292.117298] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7292.127711] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4778s workers=4 idle: 36 466 29757 [ 7292.138194] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7282s workers=2 idle: 652 [ 7296.062024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7296.068273] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7296.099094] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7296.110644] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7296.118612] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7296.125022] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7296.138951] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7296.142971] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7296.148092] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7296.151133] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7296.157922] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7296.166058] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7296.173265] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7296.184159] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7296.190990] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7296.200602] Call Trace: [ 7296.202913] [ 7296.205795] ? show_regs.cold.9+0x22/0x2f [ 7296.209311] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7296.213927] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7296.217680] ? watchdog+0x40/0x40 [ 7296.222518] ? hrtimer_interrupt+0x128/0x2c0 [ 7296.224906] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7296.228632] ? apic_timer_interrupt+0xf/0x20 [ 7296.229682] [ 7296.230217] ? native_safe_halt+0x2e/0x50 [ 7296.236217] kvm_wait+0xcd/0xe0 [ 7296.237865] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7296.243592] do_raw_spin_lock+0x177/0x190 [ 7296.258785] _raw_spin_lock+0x21/0x30 [ 7296.260228] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7296.265627] ? do_raw_spin_unlock+0x75/0x190 [ 7296.267186] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7296.275044] ? do_wait_intr+0xf0/0xf0 [ 7296.278613] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7296.284656] mdc_reint+0x2d/0x110 [mdc] [ 7296.288821] mdc_unlink+0x658/0x870 [mdc] [ 7296.290250] lmv_unlink+0x34d/0xbd0 [lmv] [ 7296.292932] ll_unlink+0x24d/0x7f0 [lustre] [ 7296.295272] vfs_unlink+0x202/0x330 [ 7296.297707] do_unlinkat+0x2fa/0x460 [ 7296.299673] __x64_sys_unlink+0x37/0x50 [ 7296.300576] do_syscall_64+0xc1/0x440 [ 7296.302656] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7296.303982] RIP: 0033:0x7f83e5f5c10b [ 7296.307062] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7296.323105] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7296.330109] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7296.332936] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7296.335300] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7296.339530] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7296.341102] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7304.059864] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 7304.084080] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7304.123694] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7304.133119] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7304.140364] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7304.148607] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7304.169243] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7304.180997] RAX: 0000000000002e0d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7304.193486] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7304.198852] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7304.203402] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7304.208770] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7304.218806] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7304.224996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7304.234765] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7304.243934] Call Trace: [ 7304.244723] [ 7304.245481] ? show_regs.cold.9+0x22/0x2f [ 7304.248648] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7304.252198] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7304.254378] ? watchdog+0x40/0x40 [ 7304.256222] ? hrtimer_interrupt+0x128/0x2c0 [ 7304.258153] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7304.260522] ? apic_timer_interrupt+0xf/0x20 [ 7304.262643] [ 7304.263585] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7304.266243] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7304.269111] do_raw_spin_lock+0x177/0x190 [ 7304.271288] _raw_spin_lock+0x21/0x30 [ 7304.273156] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7304.277528] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7304.280576] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7304.283736] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7304.287053] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7304.289753] ? woken_wake_function+0x30/0x30 [ 7304.292225] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7304.295445] kthread+0x1d1/0x200 [ 7304.298739] ? set_kthread_struct+0x70/0x70 [ 7304.304667] ret_from_fork+0x1f/0x30 [ 7316.060092] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 7316.062970] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7316.082575] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7316.087443] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7316.091334] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7316.095093] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7316.098098] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7316.107647] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7316.110160] RAX: 0000000000007153 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7316.113451] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7316.116913] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7316.120196] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7316.124151] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7316.127851] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7316.132199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7316.135568] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7316.139293] Call Trace: [ 7316.140391] [ 7316.141537] ? show_regs.cold.9+0x22/0x2f [ 7316.143305] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7316.145637] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7316.147789] ? watchdog+0x40/0x40 [ 7316.149588] ? hrtimer_interrupt+0x128/0x2c0 [ 7316.151682] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7316.154251] ? apic_timer_interrupt+0xf/0x20 [ 7316.156616] [ 7316.157637] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7316.160501] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7316.163307] do_raw_spin_lock+0x177/0x190 [ 7316.165325] _raw_spin_lock+0x21/0x30 [ 7316.167715] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7316.170205] process_one_work+0x2c8/0x700 [ 7316.172426] worker_thread+0x296/0x6e0 [ 7316.174943] ? rescuer_thread+0x570/0x570 [ 7316.176904] kthread+0x1d1/0x200 [ 7316.178580] ? set_kthread_struct+0x70/0x70 [ 7316.180524] ret_from_fork+0x1f/0x30 [ 7320.056129] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 7320.059528] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7320.127076] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7320.140881] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7320.153446] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7320.155857] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7320.184975] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7320.189864] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7320.196058] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7320.203919] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7320.207910] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7320.213123] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7320.218584] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7320.225489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7320.230036] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7320.233791] Call Trace: [ 7320.236035] [ 7320.240581] ? show_regs.cold.9+0x22/0x2f [ 7320.243474] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7320.246642] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7320.249002] ? watchdog+0x40/0x40 [ 7320.250028] ? hrtimer_interrupt+0x128/0x2c0 [ 7320.253121] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7320.255120] ? apic_timer_interrupt+0xf/0x20 [ 7320.258062] [ 7320.258936] ? native_safe_halt+0x2e/0x50 [ 7320.262867] kvm_wait+0xcd/0xe0 [ 7320.267116] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7320.272927] do_raw_spin_lock+0x177/0x190 [ 7320.279885] _raw_spin_lock+0x21/0x30 [ 7320.282713] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7320.292821] obd_mod_free+0x46/0x140 [ptlrpc] [ 7320.300951] mdc_commit_open+0x91/0xb0 [mdc] [ 7320.302957] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7320.308125] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7320.317770] after_reply+0xe2c/0x1780 [ptlrpc] [ 7320.320969] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7320.331687] ? __next_timer_interrupt+0x160/0x160 [ 7320.334924] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7320.339176] ? do_wait_intr+0xf0/0xf0 [ 7320.342958] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7320.349792] kthread+0x1d1/0x200 [ 7320.351569] ? set_kthread_struct+0x70/0x70 [ 7320.354367] ret_from_fork+0x1f/0x30 [ 7322.593087] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4809s! [ 7322.599085] Showing busy workqueues and worker pools: [ 7322.602938] workqueue events: flags=0x0 [ 7322.606417] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7322.613670] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7322.620953] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7322.624772] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7322.630459] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7322.635579] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7322.647237] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7322.652781] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7322.657446] workqueue events_long: flags=0x0 [ 7322.660010] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7322.663761] pending: lu_objects_flush [obdclass] [ 7322.667781] workqueue events_unbound: flags=0x2 [ 7322.670330] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7322.675874] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7322.679795] workqueue events_power_efficient: flags=0x80 [ 7322.685266] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7322.690852] pending: neigh_periodic_work, do_cache_clean [ 7322.695012] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7322.700803] pending: neigh_periodic_work, check_lifetime [ 7322.711010] workqueue mm_percpu_wq: flags=0x8 [ 7322.719792] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7322.724615] pending: vmstat_update, lru_add_drain_per_cpu [ 7322.734088] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7322.743682] pending: vmstat_update [ 7322.749319] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7322.754374] pending: lru_add_drain_per_cpu BAR(48) [ 7322.766407] workqueue ipv6_addrconf: flags=0x40008 [ 7322.772897] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7322.781361] pending: addrconf_verify_work [ 7322.785541] workqueue knbd0-recv: flags=0x1a [ 7322.787940] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7322.791316] in-flight: 650:recv_work [ 7322.792771] workqueue ptlrpc_pinger: flags=0x2 [ 7322.797796] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7322.805909] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7322.810961] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4809s workers=4 idle: 36 466 29757 [ 7322.814467] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7312s workers=2 idle: 652 [ 7324.062028] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 7324.067466] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7324.094693] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7324.100706] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7324.104222] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7324.107176] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7324.115428] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7324.119362] RAX: 000000000000060b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7324.122641] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7324.126347] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7324.129514] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7324.133504] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7324.136778] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7324.141425] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7324.144508] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7324.147714] Call Trace: [ 7324.149004] [ 7324.150092] ? show_regs.cold.9+0x22/0x2f [ 7324.151870] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7324.153782] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7324.156081] ? watchdog+0x40/0x40 [ 7324.157734] ? hrtimer_interrupt+0x128/0x2c0 [ 7324.159567] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7324.161893] ? apic_timer_interrupt+0xf/0x20 [ 7324.164020] [ 7324.165435] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7324.167900] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7324.170383] do_raw_spin_lock+0x177/0x190 [ 7324.172229] _raw_spin_lock+0x21/0x30 [ 7324.174012] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7324.176770] ? do_raw_spin_unlock+0x75/0x190 [ 7324.178917] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7324.181542] ? do_wait_intr+0xf0/0xf0 [ 7324.183214] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7324.185838] mdc_reint+0x2d/0x110 [mdc] [ 7324.187620] mdc_unlink+0x658/0x870 [mdc] [ 7324.189689] lmv_unlink+0x34d/0xbd0 [lmv] [ 7324.191577] ll_unlink+0x24d/0x7f0 [lustre] [ 7324.193920] vfs_unlink+0x202/0x330 [ 7324.195598] do_unlinkat+0x2fa/0x460 [ 7324.197500] __x64_sys_unlink+0x37/0x50 [ 7324.199510] do_syscall_64+0xc1/0x440 [ 7324.201784] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7324.204372] RIP: 0033:0x7f83e5f5c10b [ 7324.205936] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7324.214493] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7324.218208] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7324.221367] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7324.224943] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7324.227639] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7324.231424] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7332.066142] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 7332.078717] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7332.133178] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7332.146137] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7332.155476] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7332.168292] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7332.189494] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7332.196121] RAX: 000000000000773d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7332.203989] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7332.209369] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7332.218565] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7332.226543] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7332.230681] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7332.238733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7332.246347] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7332.251370] Call Trace: [ 7332.257244] [ 7332.258525] ? show_regs.cold.9+0x22/0x2f [ 7332.261041] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7332.266915] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7332.271610] ? watchdog+0x40/0x40 [ 7332.275960] ? hrtimer_interrupt+0x128/0x2c0 [ 7332.277513] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7332.290734] ? apic_timer_interrupt+0xf/0x20 [ 7332.294348] [ 7332.296603] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7332.307418] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7332.312942] do_raw_spin_lock+0x177/0x190 [ 7332.319447] _raw_spin_lock+0x21/0x30 [ 7332.323211] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7332.334717] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7332.343109] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7332.345178] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7332.348746] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7332.367079] ? woken_wake_function+0x30/0x30 [ 7332.368543] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7332.377187] kthread+0x1d1/0x200 [ 7332.384130] ? set_kthread_struct+0x70/0x70 [ 7332.388898] ret_from_fork+0x1f/0x30 [ 7344.060073] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 7344.062638] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7344.088459] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7344.097813] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7344.101189] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7344.104548] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7344.109371] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7344.120756] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7344.125314] RAX: 0000000000005d4e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7344.133890] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7344.137612] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7344.142421] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7344.150299] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7344.153244] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7344.156411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7344.160102] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7344.161967] Call Trace: [ 7344.162612] [ 7344.173347] ? show_regs.cold.9+0x22/0x2f [ 7344.174849] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7344.181607] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7344.183323] ? watchdog+0x40/0x40 [ 7344.185602] ? hrtimer_interrupt+0x128/0x2c0 [ 7344.190842] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7344.194750] ? apic_timer_interrupt+0xf/0x20 [ 7344.196386] [ 7344.199841] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7344.202199] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7344.204323] do_raw_spin_lock+0x177/0x190 [ 7344.211216] _raw_spin_lock+0x21/0x30 [ 7344.212811] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7344.218743] process_one_work+0x2c8/0x700 [ 7344.221784] worker_thread+0x296/0x6e0 [ 7344.225677] ? rescuer_thread+0x570/0x570 [ 7344.227139] kthread+0x1d1/0x200 [ 7344.232014] ? set_kthread_struct+0x70/0x70 [ 7344.236550] ret_from_fork+0x1f/0x30 [ 7348.059241] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 7348.065126] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7348.091117] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7348.099507] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7348.108590] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7348.112732] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7348.129792] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7348.132812] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7348.135077] RDX: 0000000000002d25 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7348.142189] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7348.145508] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7348.149811] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7348.152863] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7348.156365] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7348.159535] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7348.161828] Call Trace: [ 7348.162663] [ 7348.163800] ? show_regs.cold.9+0x22/0x2f [ 7348.165311] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7348.167106] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7348.177304] ? watchdog+0x40/0x40 [ 7348.179216] ? hrtimer_interrupt+0x128/0x2c0 [ 7348.191147] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7348.196823] ? apic_timer_interrupt+0xf/0x20 [ 7348.202180] [ 7348.202804] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7348.208994] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7348.213449] do_raw_spin_lock+0x177/0x190 [ 7348.215531] _raw_spin_lock+0x21/0x30 [ 7348.219285] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7348.224208] obd_mod_free+0x46/0x140 [ptlrpc] [ 7348.226265] mdc_commit_open+0x91/0xb0 [mdc] [ 7348.231011] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7348.240815] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7348.243348] after_reply+0xe2c/0x1780 [ptlrpc] [ 7348.246336] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7348.248795] ? __next_timer_interrupt+0x160/0x160 [ 7348.251188] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7348.253154] ? do_wait_intr+0xf0/0xf0 [ 7348.254651] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7348.256729] kthread+0x1d1/0x200 [ 7348.258066] ? set_kthread_struct+0x70/0x70 [ 7348.259445] ret_from_fork+0x1f/0x30 [ 7352.062036] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 7352.065784] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7352.113413] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7352.122133] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7352.128732] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7352.134722] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7352.144347] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7352.148736] RAX: 0000000000006ccf RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7352.152341] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7352.156510] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7352.160182] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7352.163839] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7352.167071] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7352.171080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7352.173884] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7352.177873] Call Trace: [ 7352.179377] [ 7352.180763] ? show_regs.cold.9+0x22/0x2f [ 7352.183137] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7352.186049] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7352.191472] ? watchdog+0x40/0x40 [ 7352.194173] ? hrtimer_interrupt+0x128/0x2c0 [ 7352.201320] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7352.204301] ? apic_timer_interrupt+0xf/0x20 [ 7352.207072] [ 7352.208083] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7352.211117] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7352.214008] do_raw_spin_lock+0x177/0x190 [ 7352.216214] _raw_spin_lock+0x21/0x30 [ 7352.218362] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7352.221812] ? do_raw_spin_unlock+0x75/0x190 [ 7352.224223] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7352.230823] ? do_wait_intr+0xf0/0xf0 [ 7352.235258] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7352.238977] mdc_reint+0x2d/0x110 [mdc] [ 7352.240753] mdc_unlink+0x658/0x870 [mdc] [ 7352.243754] lmv_unlink+0x34d/0xbd0 [lmv] [ 7352.248772] ll_unlink+0x24d/0x7f0 [lustre] [ 7352.252021] vfs_unlink+0x202/0x330 [ 7352.254104] do_unlinkat+0x2fa/0x460 [ 7352.256266] __x64_sys_unlink+0x37/0x50 [ 7352.258971] do_syscall_64+0xc1/0x440 [ 7352.261120] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7352.264024] RIP: 0033:0x7f83e5f5c10b [ 7352.266115] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7352.276869] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7352.285431] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7352.288709] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7352.291246] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7352.293815] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7352.296988] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7353.313882] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4840s! [ 7353.316903] Showing busy workqueues and worker pools: [ 7353.318310] workqueue events: flags=0x0 [ 7353.319437] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7353.321485] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7353.325608] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7353.328043] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7353.330767] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7353.333318] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7353.335887] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7353.338213] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7353.341304] workqueue events_long: flags=0x0 [ 7353.342805] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7353.344850] pending: lu_objects_flush [obdclass] [ 7353.346699] workqueue events_unbound: flags=0x2 [ 7353.348230] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7353.351363] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7353.354197] workqueue events_power_efficient: flags=0x80 [ 7353.357211] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7353.360409] pending: neigh_periodic_work, do_cache_clean [ 7353.363388] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7353.367427] pending: neigh_periodic_work, check_lifetime [ 7353.370211] workqueue mm_percpu_wq: flags=0x8 [ 7353.372412] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7353.378167] pending: vmstat_update, lru_add_drain_per_cpu [ 7353.381847] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7353.385358] pending: vmstat_update [ 7353.387230] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7353.390482] pending: lru_add_drain_per_cpu BAR(48) [ 7353.392996] workqueue ipv6_addrconf: flags=0x40008 [ 7353.395142] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7353.397807] pending: addrconf_verify_work [ 7353.400138] workqueue knbd0-recv: flags=0x1a [ 7353.402423] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7353.405664] in-flight: 650:recv_work [ 7353.407759] workqueue ptlrpc_pinger: flags=0x2 [ 7353.410216] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7353.413182] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7353.416592] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4840s workers=4 idle: 36 466 29757 [ 7353.420971] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7343s workers=2 idle: 652 [ 7360.059028] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 7360.062576] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7360.084347] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7360.091926] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7360.098873] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7360.101617] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7360.111167] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7360.116478] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7360.120064] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7360.124013] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7360.127303] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7360.131476] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7360.135160] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7360.139901] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7360.143141] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7360.147044] Call Trace: [ 7360.148464] [ 7360.149430] ? show_regs.cold.9+0x22/0x2f [ 7360.151435] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7360.153225] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7360.155316] ? watchdog+0x40/0x40 [ 7360.157344] ? hrtimer_interrupt+0x128/0x2c0 [ 7360.159792] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7360.161980] ? apic_timer_interrupt+0xf/0x20 [ 7360.164368] [ 7360.165416] ? native_safe_halt+0x2e/0x50 [ 7360.167647] kvm_wait+0xcd/0xe0 [ 7360.169486] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7360.172325] do_raw_spin_lock+0x177/0x190 [ 7360.175062] _raw_spin_lock+0x21/0x30 [ 7360.176666] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7360.180535] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7360.184703] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7360.188275] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7360.192631] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7360.207688] ? woken_wake_function+0x30/0x30 [ 7360.209990] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7360.214651] kthread+0x1d1/0x200 [ 7360.217374] ? set_kthread_struct+0x70/0x70 [ 7360.219512] ret_from_fork+0x1f/0x30 [ 7372.060729] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 7372.064621] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7372.085074] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7372.091295] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7372.096026] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7372.099704] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7372.101743] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7372.110003] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7372.115008] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7372.119231] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7372.123608] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7372.129016] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7372.133515] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7372.136900] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7372.140925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7372.143809] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7372.147422] Call Trace: [ 7372.148612] [ 7372.149905] ? show_regs.cold.9+0x22/0x2f [ 7372.151811] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7372.154780] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7372.156814] ? watchdog+0x40/0x40 [ 7372.158389] ? hrtimer_interrupt+0x128/0x2c0 [ 7372.160325] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7372.163749] ? apic_timer_interrupt+0xf/0x20 [ 7372.165871] [ 7372.166926] ? native_safe_halt+0x2e/0x50 [ 7372.168922] kvm_wait+0xcd/0xe0 [ 7372.170528] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7372.172960] do_raw_spin_lock+0x177/0x190 [ 7372.174745] _raw_spin_lock+0x21/0x30 [ 7372.177227] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7372.180573] process_one_work+0x2c8/0x700 [ 7372.183099] worker_thread+0x296/0x6e0 [ 7372.185806] ? rescuer_thread+0x570/0x570 [ 7372.187620] kthread+0x1d1/0x200 [ 7372.189906] ? set_kthread_struct+0x70/0x70 [ 7372.191821] ret_from_fork+0x1f/0x30 [ 7380.058252] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 7380.060383] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7380.062017] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 7380.095030] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7380.098960] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7380.099047] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7380.099055] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7380.099057] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7380.099072] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7380.099077] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7380.099081] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7380.099084] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7380.099086] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7380.099088] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7380.099091] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7380.099094] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7380.099098] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7380.099100] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7380.099110] Call Trace: [ 7380.099116] [ 7380.099123] ? show_regs.cold.9+0x22/0x2f [ 7380.099133] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7380.099139] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7380.099143] ? watchdog+0x40/0x40 [ 7380.099147] ? hrtimer_interrupt+0x128/0x2c0 [ 7380.099152] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7380.099158] ? apic_timer_interrupt+0xf/0x20 [ 7380.099163] [ 7380.099165] ? native_safe_halt+0x2e/0x50 [ 7380.099170] kvm_wait+0xcd/0xe0 [ 7380.099178] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7380.099185] do_raw_spin_lock+0x177/0x190 [ 7380.099189] _raw_spin_lock+0x21/0x30 [ 7380.099195] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7380.099666] ? do_raw_spin_unlock+0x75/0x190 [ 7380.099672] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7380.122337] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7380.148962] ? do_wait_intr+0xf0/0xf0 [ 7380.148984] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7380.149248] mdc_reint+0x2d/0x110 [mdc] [ 7380.149284] mdc_unlink+0x658/0x870 [mdc] [ 7380.149310] lmv_unlink+0x34d/0xbd0 [lmv] [ 7380.149341] ll_unlink+0x24d/0x7f0 [lustre] [ 7380.159585] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7380.159608] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7380.159612] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7380.159618] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7380.159621] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7380.159623] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7380.159625] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7380.159628] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7380.159630] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7380.159633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7380.159636] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7380.159644] Call Trace: [ 7380.159648] [ 7380.159653] ? show_regs.cold.9+0x22/0x2f [ 7380.159661] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7380.159667] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7380.159671] ? watchdog+0x40/0x40 [ 7380.159675] ? hrtimer_interrupt+0x128/0x2c0 [ 7380.159679] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7380.164232] vfs_unlink+0x202/0x330 [ 7380.166367] ? apic_timer_interrupt+0xf/0x20 [ 7380.175742] do_unlinkat+0x2fa/0x460 [ 7380.179129] [ 7380.182345] __x64_sys_unlink+0x37/0x50 [ 7380.186895] ? native_safe_halt+0x2e/0x50 [ 7380.189228] do_syscall_64+0xc1/0x440 [ 7380.189245] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7380.189254] RIP: 0033:0x7f83e5f5c10b [ 7380.189261] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7380.189266] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7380.189272] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7380.189274] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7380.189277] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7380.189279] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7380.189282] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7380.543499] kvm_wait+0xcd/0xe0 [ 7380.545374] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7380.550228] do_raw_spin_lock+0x177/0x190 [ 7380.552233] _raw_spin_lock+0x21/0x30 [ 7380.553905] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7380.555917] obd_mod_free+0x46/0x140 [ptlrpc] [ 7380.565213] mdc_commit_open+0x91/0xb0 [mdc] [ 7380.567754] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7380.587518] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7380.589687] after_reply+0xe2c/0x1780 [ptlrpc] [ 7380.603632] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7380.618209] ? __next_timer_interrupt+0x160/0x160 [ 7380.624223] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7380.631106] ? do_wait_intr+0xf0/0xf0 [ 7380.633973] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7380.637909] kthread+0x1d1/0x200 [ 7380.639863] ? set_kthread_struct+0x70/0x70 [ 7380.641506] ret_from_fork+0x1f/0x30 [ 7383.521428] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4870s! [ 7383.525908] Showing busy workqueues and worker pools: [ 7383.528206] workqueue events: flags=0x0 [ 7383.530674] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7383.533844] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7383.538088] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7383.541113] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7383.547113] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7383.551172] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7383.554303] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7383.557457] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7383.562207] workqueue events_long: flags=0x0 [ 7383.563487] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7383.567194] pending: lu_objects_flush [obdclass] [ 7383.569231] workqueue events_unbound: flags=0x2 [ 7383.572850] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7383.575733] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7383.578277] workqueue events_power_efficient: flags=0x80 [ 7383.583351] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7383.586441] pending: neigh_periodic_work, do_cache_clean [ 7383.589660] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7383.593433] pending: neigh_periodic_work, check_lifetime [ 7383.596488] workqueue mm_percpu_wq: flags=0x8 [ 7383.598804] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7383.601632] pending: vmstat_update, lru_add_drain_per_cpu [ 7383.603389] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7383.605802] pending: vmstat_update [ 7383.607734] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7383.609749] pending: lru_add_drain_per_cpu BAR(48) [ 7383.611474] workqueue ipv6_addrconf: flags=0x40008 [ 7383.613419] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7383.616100] pending: addrconf_verify_work [ 7383.617929] workqueue knbd0-recv: flags=0x1a [ 7383.619754] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7383.622796] in-flight: 650:recv_work [ 7383.624437] workqueue ptlrpc_pinger: flags=0x2 [ 7383.626254] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7383.628871] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7383.631519] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4870s workers=4 idle: 36 466 29757 [ 7383.634717] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7373s workers=2 idle: 652 [ 7388.059476] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 7388.062276] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7388.116779] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7388.131105] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7388.135663] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7388.140251] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7388.149919] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7388.153394] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7388.156471] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7388.160335] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7388.163787] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7388.167111] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7388.170171] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7388.173913] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7388.178597] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7388.182151] Call Trace: [ 7388.183386] [ 7388.186762] ? show_regs.cold.9+0x22/0x2f [ 7388.190255] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7388.192225] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7388.194928] ? watchdog+0x40/0x40 [ 7388.196263] ? hrtimer_interrupt+0x128/0x2c0 [ 7388.198355] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7388.200305] ? apic_timer_interrupt+0xf/0x20 [ 7388.202442] [ 7388.203336] ? native_safe_halt+0x2e/0x50 [ 7388.205311] kvm_wait+0xcd/0xe0 [ 7388.206954] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7388.209133] do_raw_spin_lock+0x177/0x190 [ 7388.210920] _raw_spin_lock+0x21/0x30 [ 7388.212390] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7388.215127] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7388.217908] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7388.220837] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7388.227174] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7388.230897] ? woken_wake_function+0x30/0x30 [ 7388.233031] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7388.239848] kthread+0x1d1/0x200 [ 7388.241838] ? set_kthread_struct+0x70/0x70 [ 7388.244122] ret_from_fork+0x1f/0x30 [ 7400.060497] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 7400.071841] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7400.131761] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7400.139273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7400.150979] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7400.158929] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7400.166803] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7400.180887] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7400.188851] RAX: 0000000000004f5e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7400.195415] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7400.210828] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7400.221938] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7400.227290] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7400.233586] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7400.237732] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7400.242474] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7400.244766] Call Trace: [ 7400.246226] [ 7400.248331] ? show_regs.cold.9+0x22/0x2f [ 7400.252342] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7400.254337] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7400.261450] ? watchdog+0x40/0x40 [ 7400.262725] ? hrtimer_interrupt+0x128/0x2c0 [ 7400.264113] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7400.266968] ? apic_timer_interrupt+0xf/0x20 [ 7400.269603] [ 7400.270444] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7400.277280] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7400.283756] do_raw_spin_lock+0x177/0x190 [ 7400.286044] _raw_spin_lock+0x21/0x30 [ 7400.289572] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7400.294004] process_one_work+0x2c8/0x700 [ 7400.297243] worker_thread+0x296/0x6e0 [ 7400.299822] ? rescuer_thread+0x570/0x570 [ 7400.301256] kthread+0x1d1/0x200 [ 7400.302751] ? set_kthread_struct+0x70/0x70 [ 7400.305742] ret_from_fork+0x1f/0x30 [ 7408.058352] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 7408.062081] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 7408.066626] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) [ 7408.070203] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7408.070289] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7408.070296] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7408.070298] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7408.070317] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7408.070322] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7408.070326] RAX: 0000000000005908 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7408.070329] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7408.070331] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7408.070334] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7408.070336] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7408.070340] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7408.070347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7408.070349] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7408.070356] Call Trace: [ 7408.070365] [ 7408.070371] ? show_regs.cold.9+0x22/0x2f [ 7408.070380] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7408.070390] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7408.070396] ? watchdog+0x40/0x40 [ 7408.070401] ? hrtimer_interrupt+0x128/0x2c0 [ 7408.070405] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7408.070417] ? apic_timer_interrupt+0xf/0x20 [ 7408.070423] [ 7408.070425] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7408.070434] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7408.070440] do_raw_spin_lock+0x177/0x190 [ 7408.070445] _raw_spin_lock+0x21/0x30 [ 7408.070451] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7408.070776] ? do_raw_spin_unlock+0x75/0x190 [ 7408.070782] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7408.070981] ? do_wait_intr+0xf0/0xf0 [ 7408.070989] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7408.071193] mdc_reint+0x2d/0x110 [mdc] [ 7408.071237] mdc_unlink+0x658/0x870 [mdc] [ 7408.071266] lmv_unlink+0x34d/0xbd0 [lmv] [ 7408.071295] ll_unlink+0x24d/0x7f0 [lustre] [ 7408.071427] vfs_unlink+0x202/0x330 [ 7408.071439] do_unlinkat+0x2fa/0x460 [ 7408.071444] __x64_sys_unlink+0x37/0x50 [ 7408.071448] do_syscall_64+0xc1/0x440 [ 7408.071454] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7408.071461] RIP: 0033:0x7f83e5f5c10b [ 7408.071467] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7408.071471] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7408.071475] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7408.071477] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7408.071479] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7408.071482] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7408.071484] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7408.247080] lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7408.282903] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7408.287662] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7408.290834] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7408.292813] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7408.308830] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7408.312889] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7408.315582] RDX: 0000000000002caa RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7408.319692] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7408.324686] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7408.329184] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7408.336345] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7408.345028] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7408.350314] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7408.352858] Call Trace: [ 7408.353670] [ 7408.354461] ? show_regs.cold.9+0x22/0x2f [ 7408.355997] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7408.358394] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7408.360337] ? watchdog+0x40/0x40 [ 7408.361809] ? hrtimer_interrupt+0x128/0x2c0 [ 7408.364107] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7408.367790] ? apic_timer_interrupt+0xf/0x20 [ 7408.370992] [ 7408.371936] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7408.374928] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7408.377634] do_raw_spin_lock+0x177/0x190 [ 7408.379582] _raw_spin_lock+0x21/0x30 [ 7408.382639] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7408.386182] obd_mod_free+0x46/0x140 [ptlrpc] [ 7408.391110] mdc_commit_open+0x91/0xb0 [mdc] [ 7408.394029] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7408.396857] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7408.404003] after_reply+0xe2c/0x1780 [ptlrpc] [ 7408.407782] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7408.411135] ? __next_timer_interrupt+0x160/0x160 [ 7408.413659] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7408.416138] ? do_wait_intr+0xf0/0xf0 [ 7408.418097] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7408.421419] kthread+0x1d1/0x200 [ 7408.423025] ? set_kthread_struct+0x70/0x70 [ 7408.425683] ret_from_fork+0x1f/0x30 [ 7413.732829] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4900s! [ 7413.740591] Showing busy workqueues and worker pools: [ 7413.743729] workqueue events: flags=0x0 [ 7413.745182] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7413.751242] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7413.764692] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7413.778477] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7413.785024] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7413.792188] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7413.803866] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7413.808846] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7413.819044] workqueue events_long: flags=0x0 [ 7413.827474] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7413.831807] pending: lu_objects_flush [obdclass] [ 7413.836724] workqueue events_unbound: flags=0x2 [ 7413.839114] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7413.845708] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7413.854106] workqueue events_power_efficient: flags=0x80 [ 7413.856116] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7413.864144] pending: neigh_periodic_work, do_cache_clean [ 7413.869094] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7413.871936] pending: neigh_periodic_work, check_lifetime [ 7413.877900] workqueue mm_percpu_wq: flags=0x8 [ 7413.886428] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7413.891496] pending: vmstat_update, lru_add_drain_per_cpu [ 7413.897673] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7413.901017] pending: vmstat_update [ 7413.902814] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7413.906013] pending: lru_add_drain_per_cpu BAR(48) [ 7413.908795] workqueue ipv6_addrconf: flags=0x40008 [ 7413.911093] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7413.914454] pending: addrconf_verify_work [ 7413.918726] workqueue knbd0-recv: flags=0x1a [ 7413.921371] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7413.924761] in-flight: 650:recv_work [ 7413.928050] workqueue ptlrpc_pinger: flags=0x2 [ 7413.929599] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7413.933930] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7413.937040] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4900s workers=4 idle: 36 466 29757 [ 7413.939808] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7403s workers=2 idle: 652 [ 7416.063199] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 7416.068573] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7416.100859] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7416.109568] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7416.112767] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7416.114787] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7416.124770] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7416.131414] RAX: 0000000000007bad RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7416.135514] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7416.138450] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7416.144207] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7416.147550] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7416.150458] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7416.156652] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7416.161653] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7416.164024] Call Trace: [ 7416.166743] [ 7416.168436] ? show_regs.cold.9+0x22/0x2f [ 7416.169687] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7416.171775] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7416.176182] ? watchdog+0x40/0x40 [ 7416.177700] ? hrtimer_interrupt+0x128/0x2c0 [ 7416.180206] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7416.181978] ? apic_timer_interrupt+0xf/0x20 [ 7416.183671] [ 7416.184308] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7416.193237] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7416.195112] do_raw_spin_lock+0x177/0x190 [ 7416.196763] _raw_spin_lock+0x21/0x30 [ 7416.198196] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7416.205620] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7416.208967] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7416.214135] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7416.217608] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7416.222160] ? woken_wake_function+0x30/0x30 [ 7416.224160] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7416.227427] kthread+0x1d1/0x200 [ 7416.232810] ? set_kthread_struct+0x70/0x70 [ 7416.235470] ret_from_fork+0x1f/0x30 [ 7428.060010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 7428.062409] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7428.078759] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7428.081834] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7428.084615] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7428.086751] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 7428.088963] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 7428.094633] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7428.096755] RAX: 0000000000004546 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7428.099202] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7428.101971] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7428.103866] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7428.106399] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7428.109360] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7428.112473] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7428.115735] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7428.119736] Call Trace: [ 7428.121723] [ 7428.123852] ? show_regs.cold.9+0x22/0x2f [ 7428.126889] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7428.130004] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7428.132842] ? watchdog+0x40/0x40 [ 7428.134768] ? hrtimer_interrupt+0x128/0x2c0 [ 7428.135849] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7428.138180] ? apic_timer_interrupt+0xf/0x20 [ 7428.139817] [ 7428.141255] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 7428.145884] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7428.149918] do_raw_spin_lock+0x177/0x190 [ 7428.151866] _raw_spin_lock+0x21/0x30 [ 7428.153957] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7428.156357] process_one_work+0x2c8/0x700 [ 7428.157907] worker_thread+0x296/0x6e0 [ 7428.159475] ? rescuer_thread+0x570/0x570 [ 7428.161047] kthread+0x1d1/0x200 [ 7428.162668] ? set_kthread_struct+0x70/0x70 [ 7428.164037] ret_from_fork+0x1f/0x30 [ 7436.063265] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7436.084229] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7436.117203] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7436.124656] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7436.129899] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7436.134516] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7436.155409] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7436.162174] RAX: 0000000000007aa9 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7436.173103] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7436.182578] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7436.192225] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7436.199766] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7436.206997] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7436.214720] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7436.221436] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7436.229184] Call Trace: [ 7436.234244] [ 7436.235376] ? show_regs.cold.9+0x22/0x2f [ 7436.240216] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7436.243408] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7436.246285] ? watchdog+0x40/0x40 [ 7436.248359] ? hrtimer_interrupt+0x128/0x2c0 [ 7436.250508] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7436.253622] ? apic_timer_interrupt+0xf/0x20 [ 7436.256371] [ 7436.257555] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7436.261329] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7436.264719] do_raw_spin_lock+0x177/0x190 [ 7436.267303] _raw_spin_lock+0x21/0x30 [ 7436.269349] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7436.273165] ? do_raw_spin_unlock+0x75/0x190 [ 7436.275552] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7436.278738] ? do_wait_intr+0xf0/0xf0 [ 7436.281486] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7436.284793] mdc_reint+0x2d/0x110 [mdc] [ 7436.287354] mdc_unlink+0x658/0x870 [mdc] [ 7436.290637] lmv_unlink+0x34d/0xbd0 [lmv] [ 7436.294142] ll_unlink+0x24d/0x7f0 [lustre] [ 7436.297259] vfs_unlink+0x202/0x330 [ 7436.305123] do_unlinkat+0x2fa/0x460 [ 7436.307309] __x64_sys_unlink+0x37/0x50 [ 7436.309484] do_syscall_64+0xc1/0x440 [ 7436.312152] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7436.317728] RIP: 0033:0x7f83e5f5c10b [ 7436.320767] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7436.337398] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7436.347557] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7436.359672] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7436.374515] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7436.383530] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7436.390132] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7440.056274] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 7440.060507] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7440.106383] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7440.129846] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7440.146176] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7440.158870] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7440.181268] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7440.190347] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7440.201915] RDX: 0000000000000dcf RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7440.214948] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7440.228201] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7440.235271] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7440.256389] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7440.262186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7440.272137] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7440.280912] Call Trace: [ 7440.281942] [ 7440.282707] ? show_regs.cold.9+0x22/0x2f [ 7440.287991] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7440.303050] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7440.310350] ? watchdog+0x40/0x40 [ 7440.312475] ? hrtimer_interrupt+0x128/0x2c0 [ 7440.321184] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7440.328959] ? apic_timer_interrupt+0xf/0x20 [ 7440.334875] [ 7440.337926] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7440.344837] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7440.360066] do_raw_spin_lock+0x177/0x190 [ 7440.367311] _raw_spin_lock+0x21/0x30 [ 7440.376363] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7440.384602] obd_mod_free+0x46/0x140 [ptlrpc] [ 7440.390215] mdc_commit_open+0x91/0xb0 [mdc] [ 7440.394480] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7440.402813] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7440.409669] after_reply+0xe2c/0x1780 [ptlrpc] [ 7440.415217] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7440.424710] ? __next_timer_interrupt+0x160/0x160 [ 7440.428651] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7440.433390] ? do_wait_intr+0xf0/0xf0 [ 7440.436838] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7440.441426] kthread+0x1d1/0x200 [ 7440.443361] ? set_kthread_struct+0x70/0x70 [ 7440.446973] ret_from_fork+0x1f/0x30 [ 7441.084627] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 7441.091591] rcu: 0-....: (3475827 ticks this GP) idle=30a/1/0x4000000000000000 softirq=128519/128519 fqs=126395 [ 7441.108640] (detected by 1, t=5460125 jiffies, g=262441, q=58751) [ 7441.111851] Sending NMI from CPU 1 to CPUs 0: [ 7441.121376] NMI backtrace for cpu 0 [ 7441.121384] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7441.121386] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7441.121388] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7441.121390] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7441.121392] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 [ 7441.121395] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7441.121396] RDX: 0000000000007e15 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7441.121398] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7441.121399] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7441.121400] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7441.121402] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7441.121403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7441.121404] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7441.121406] Call Trace: [ 7441.121406] [ 7441.121407] ? show_regs.cold.9+0x22/0x2f [ 7441.121408] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7441.121409] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7441.121410] ? nmi_handle+0xbd/0x210 [ 7441.121411] ? default_do_nmi+0x4b/0x1b0 [ 7441.121412] ? do_nmi+0x280/0x380 [ 7441.121413] ? end_repeat_nmi+0x16/0x1f [ 7441.121414] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7441.121416] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7441.121417] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7441.121418] [ 7441.121419] do_raw_spin_lock+0x177/0x190 [ 7441.121420] _raw_spin_lock+0x21/0x30 [ 7441.121421] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7441.121422] obd_mod_free+0x46/0x140 [ptlrpc] [ 7441.121423] mdc_commit_open+0x91/0xb0 [mdc] [ 7441.121424] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7441.121425] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7441.121427] after_reply+0xe2c/0x1780 [ptlrpc] [ 7441.121428] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7441.121429] ? __next_timer_interrupt+0x160/0x160 [ 7441.121431] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7441.121432] ? do_wait_intr+0xf0/0xf0 [ 7441.121433] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7441.121434] kthread+0x1d1/0x200 [ 7441.121435] ? set_kthread_struct+0x70/0x70 [ 7441.121436] ret_from_fork+0x1f/0x30 [ 7441.121894] rcu: rcu_sched kthread starved for 4929312 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 7441.376821] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7441.387558] rcu: RCU grace-period kthread stack dump: [ 7441.392954] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7441.401250] Call Trace: [ 7441.402841] __schedule+0x351/0xcb0 [ 7441.405167] schedule+0xc0/0x180 [ 7441.408489] schedule_timeout+0xb4/0x190 [ 7441.411393] ? do_raw_spin_unlock+0x75/0x190 [ 7441.414034] ? __next_timer_interrupt+0x160/0x160 [ 7441.416595] ? prepare_to_swait_event+0xf3/0x140 [ 7441.419744] rcu_gp_kthread+0x8dd/0x15b0 [ 7441.422883] ? force_qs_rnp+0x2c0/0x2c0 [ 7441.425888] kthread+0x1d1/0x200 [ 7441.428036] ? set_kthread_struct+0x70/0x70 [ 7441.430610] ret_from_fork+0x1f/0x30 [ 7441.436727] rcu: Stack dump where RCU GP kthread last ran: [ 7441.444255] Sending NMI from CPU 1 to CPUs 2: [ 7441.450252] NMI backtrace for cpu 2 [ 7441.450259] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7441.450403] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7441.450405] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7441.450412] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7441.450414] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7441.450416] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 [ 7441.450418] RAX: 000000000000170f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7441.450420] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7441.450421] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7441.450423] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7441.450424] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7441.450426] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7441.450427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7441.450429] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7441.450430] Call Trace: [ 7441.450431] [ 7441.450432] ? show_regs.cold.9+0x22/0x2f [ 7441.450433] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7441.450435] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7441.450436] ? nmi_handle+0xbd/0x210 [ 7441.450437] ? default_do_nmi+0x4b/0x1b0 [ 7441.450443] ? do_nmi+0x280/0x380 [ 7441.450444] ? end_repeat_nmi+0x16/0x1f [ 7441.450445] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7441.450446] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7441.450447] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7441.450448] [ 7441.450449] do_raw_spin_lock+0x177/0x190 [ 7441.450450] _raw_spin_lock+0x21/0x30 [ 7441.450451] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7441.450452] process_one_work+0x2c8/0x700 [ 7441.450453] worker_thread+0x296/0x6e0 [ 7441.450454] ? rescuer_thread+0x570/0x570 [ 7441.450456] kthread+0x1d1/0x200 [ 7441.450457] ? set_kthread_struct+0x70/0x70 [ 7441.450458] ret_from_fork+0x1f/0x30 [ 7444.458570] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4931s! [ 7444.475656] Showing busy workqueues and worker pools: [ 7444.477404] workqueue events: flags=0x0 [ 7444.488381] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7444.490611] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7444.518780] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7444.520989] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7444.537134] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7444.546497] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7444.559304] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7444.570491] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7444.576819] workqueue events_long: flags=0x0 [ 7444.583273] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7444.593751] pending: lu_objects_flush [obdclass] [ 7444.606593] workqueue events_unbound: flags=0x2 [ 7444.616967] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7444.619195] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7444.629015] workqueue events_power_efficient: flags=0x80 [ 7444.639496] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7444.644909] pending: neigh_periodic_work, do_cache_clean [ 7444.654489] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7444.665020] pending: neigh_periodic_work, check_lifetime [ 7444.668598] workqueue mm_percpu_wq: flags=0x8 [ 7444.671056] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7444.676360] pending: vmstat_update, lru_add_drain_per_cpu [ 7444.679154] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7444.681511] pending: vmstat_update [ 7444.692629] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7444.703770] pending: lru_add_drain_per_cpu BAR(48) [ 7444.708111] workqueue ipv6_addrconf: flags=0x40008 [ 7444.712280] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7444.722876] pending: addrconf_verify_work [ 7444.725404] workqueue knbd0-recv: flags=0x1a [ 7444.728106] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7444.731865] in-flight: 650:recv_work [ 7444.734022] workqueue ptlrpc_pinger: flags=0x2 [ 7444.736613] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7444.740682] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7444.746303] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4931s workers=4 idle: 36 466 29757 [ 7444.763728] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7434s workers=2 idle: 652 [ 7464.062063] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7464.063996] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7464.079433] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7464.083402] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7464.087109] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7464.089815] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7464.098124] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7464.100995] RAX: 0000000000004d13 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7464.103971] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7464.106270] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7464.109017] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7464.111904] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7464.114795] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7464.117652] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7464.119836] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7464.122043] Call Trace: [ 7464.122934] [ 7464.123571] ? show_regs.cold.9+0x22/0x2f [ 7464.124936] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7464.127584] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7464.129163] ? watchdog+0x40/0x40 [ 7464.130425] ? hrtimer_interrupt+0x128/0x2c0 [ 7464.131806] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7464.133765] ? apic_timer_interrupt+0xf/0x20 [ 7464.135256] [ 7464.136207] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7464.138010] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7464.140924] do_raw_spin_lock+0x177/0x190 [ 7464.142149] _raw_spin_lock+0x21/0x30 [ 7464.143234] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7464.145196] ? do_raw_spin_unlock+0x75/0x190 [ 7464.146717] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7464.148690] ? do_wait_intr+0xf0/0xf0 [ 7464.150092] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7464.152505] mdc_reint+0x2d/0x110 [mdc] [ 7464.154044] mdc_unlink+0x658/0x870 [mdc] [ 7464.155366] lmv_unlink+0x34d/0xbd0 [lmv] [ 7464.156733] ll_unlink+0x24d/0x7f0 [lustre] [ 7464.159051] vfs_unlink+0x202/0x330 [ 7464.160249] do_unlinkat+0x2fa/0x460 [ 7464.161563] __x64_sys_unlink+0x37/0x50 [ 7464.162736] do_syscall_64+0xc1/0x440 [ 7464.164200] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7464.165730] RIP: 0033:0x7f83e5f5c10b [ 7464.167580] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7464.173391] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7464.176093] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7464.178227] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7464.181255] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7464.183665] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7464.186246] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7468.059679] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 7468.060351] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 7468.066247] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7468.066330] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7468.066338] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7468.066341] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7468.066354] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7468.066359] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7468.066364] RAX: 0000000000005005 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7468.066367] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7468.066369] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7468.066371] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7468.066373] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7468.066376] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7468.066379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7468.066381] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7468.066388] Call Trace: [ 7468.066394] [ 7468.066400] ? show_regs.cold.9+0x22/0x2f [ 7468.071164] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7468.071240] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7468.071247] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7468.071254] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7468.091639] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7468.096470] [ 7468.096479] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7468.096496] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7468.096500] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7468.096517] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7468.096519] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7468.096522] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7468.096524] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7468.096527] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7468.096533] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7468.096536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7468.096539] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7468.096548] Call Trace: [ 7468.096553] [ 7468.096559] ? show_regs.cold.9+0x22/0x2f [ 7468.096569] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7468.096575] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7468.096580] ? watchdog+0x40/0x40 [ 7468.096584] ? hrtimer_interrupt+0x128/0x2c0 [ 7468.096587] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7468.096594] ? apic_timer_interrupt+0xf/0x20 [ 7468.096600] [ 7468.096603] ? native_safe_halt+0x2e/0x50 [ 7468.096607] kvm_wait+0xcd/0xe0 [ 7468.096614] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7468.096625] do_raw_spin_lock+0x177/0x190 [ 7468.096629] _raw_spin_lock+0x21/0x30 [ 7468.096633] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7468.100569] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7468.100586] ? watchdog+0x40/0x40 [ 7468.100593] ? hrtimer_interrupt+0x128/0x2c0 [ 7468.100597] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7468.100605] ? apic_timer_interrupt+0xf/0x20 [ 7468.100610] [ 7468.100615] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7468.100622] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7468.100629] do_raw_spin_lock+0x177/0x190 [ 7468.100635] _raw_spin_lock+0x21/0x30 [ 7468.100640] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7468.101082] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7468.103516] process_one_work+0x2c8/0x700 [ 7468.103541] worker_thread+0x296/0x6e0 [ 7468.103545] ? rescuer_thread+0x570/0x570 [ 7468.103549] kthread+0x1d1/0x200 [ 7468.103553] ? set_kthread_struct+0x70/0x70 [ 7468.103557] ret_from_fork+0x1f/0x30 [ 7468.246000] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7468.247852] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7468.249798] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7468.251997] ? woken_wake_function+0x30/0x30 [ 7468.253444] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7468.255379] kthread+0x1d1/0x200 [ 7468.256594] ? set_kthread_struct+0x70/0x70 [ 7468.257996] ret_from_fork+0x1f/0x30 [ 7472.060640] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 7472.071463] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7472.133417] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7472.143880] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7472.150331] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7472.154077] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7472.162611] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7472.166485] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7472.169188] RDX: 0000000000007810 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7472.174281] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7472.177858] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7472.180783] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7472.185040] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7472.188512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7472.191136] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7472.194113] Call Trace: [ 7472.195269] [ 7472.196331] ? show_regs.cold.9+0x22/0x2f [ 7472.198083] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7472.200434] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7472.202285] ? watchdog+0x40/0x40 [ 7472.203985] ? hrtimer_interrupt+0x128/0x2c0 [ 7472.205644] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7472.207758] ? apic_timer_interrupt+0xf/0x20 [ 7472.209352] [ 7472.210222] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7472.212663] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7472.216547] do_raw_spin_lock+0x177/0x190 [ 7472.220723] _raw_spin_lock+0x21/0x30 [ 7472.229217] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7472.234745] obd_mod_free+0x46/0x140 [ptlrpc] [ 7472.239528] mdc_commit_open+0x91/0xb0 [mdc] [ 7472.244303] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7472.246770] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7472.255307] after_reply+0xe2c/0x1780 [ptlrpc] [ 7472.258739] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7472.264211] ? __next_timer_interrupt+0x160/0x160 [ 7472.268544] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7472.274254] ? do_wait_intr+0xf0/0xf0 [ 7472.279648] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7472.288346] kthread+0x1d1/0x200 [ 7472.291849] ? set_kthread_struct+0x70/0x70 [ 7472.295009] ret_from_fork+0x1f/0x30 [ 7475.176644] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4961s! [ 7475.180294] Showing busy workqueues and worker pools: [ 7475.183355] workqueue events: flags=0x0 [ 7475.188447] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7475.198332] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7475.211152] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7475.221096] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7475.230934] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7475.236756] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7475.244157] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7475.256983] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7475.268496] workqueue events_long: flags=0x0 [ 7475.274125] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7475.282350] pending: lu_objects_flush [obdclass] [ 7475.286868] workqueue events_unbound: flags=0x2 [ 7475.291745] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7475.300049] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7475.307333] workqueue events_power_efficient: flags=0x80 [ 7475.316631] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7475.322333] pending: neigh_periodic_work, do_cache_clean [ 7475.326787] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7475.332457] pending: neigh_periodic_work, check_lifetime [ 7475.335780] workqueue mm_percpu_wq: flags=0x8 [ 7475.339487] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7475.346884] pending: vmstat_update, lru_add_drain_per_cpu [ 7475.349010] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7475.357267] pending: vmstat_update [ 7475.364617] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7475.371175] pending: lru_add_drain_per_cpu BAR(48) [ 7475.376138] workqueue ipv6_addrconf: flags=0x40008 [ 7475.379085] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7475.386209] pending: addrconf_verify_work [ 7475.389909] workqueue knbd0-recv: flags=0x1a [ 7475.393023] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7475.395947] in-flight: 650:recv_work [ 7475.397808] workqueue ptlrpc_pinger: flags=0x2 [ 7475.400390] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7475.408229] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7475.418276] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4962s workers=4 idle: 36 466 29757 [ 7475.427067] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7465s workers=2 idle: 652 [ 7492.062125] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7492.065288] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7492.093389] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7492.099944] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7492.105154] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7492.108144] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7492.118758] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7492.122330] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7492.125044] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7492.128369] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7492.135108] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7492.141748] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7492.152851] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7492.164134] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7492.170265] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7492.174949] Call Trace: [ 7492.175822] [ 7492.177367] ? show_regs.cold.9+0x22/0x2f [ 7492.181919] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7492.185324] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7492.190104] ? watchdog+0x40/0x40 [ 7492.192177] ? hrtimer_interrupt+0x128/0x2c0 [ 7492.196449] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7492.201307] ? apic_timer_interrupt+0xf/0x20 [ 7492.206641] [ 7492.209210] ? native_safe_halt+0x2e/0x50 [ 7492.214235] kvm_wait+0xcd/0xe0 [ 7492.216155] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7492.220184] do_raw_spin_lock+0x177/0x190 [ 7492.223145] _raw_spin_lock+0x21/0x30 [ 7492.225313] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7492.229090] ? do_raw_spin_unlock+0x75/0x190 [ 7492.231454] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7492.237063] ? do_wait_intr+0xf0/0xf0 [ 7492.238346] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7492.245301] mdc_reint+0x2d/0x110 [mdc] [ 7492.246790] mdc_unlink+0x658/0x870 [mdc] [ 7492.248255] lmv_unlink+0x34d/0xbd0 [lmv] [ 7492.257558] ll_unlink+0x24d/0x7f0 [lustre] [ 7492.263544] vfs_unlink+0x202/0x330 [ 7492.267324] do_unlinkat+0x2fa/0x460 [ 7492.272793] __x64_sys_unlink+0x37/0x50 [ 7492.276656] do_syscall_64+0xc1/0x440 [ 7492.279830] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7492.284633] RIP: 0033:0x7f83e5f5c10b [ 7492.287068] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7492.301504] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7492.304894] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7492.330069] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7492.340362] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7492.348447] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7492.355391] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7496.059698] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 7496.061616] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 7496.071365] Modules linked in: [ 7496.074365] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7496.074589] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7496.074596] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7496.074602] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7496.074872] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7496.074900] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7496.074905] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7496.074909] RAX: 0000000000007fb8 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7496.074911] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7496.074914] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7496.074916] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7496.074918] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7496.074921] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7496.074923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7496.074926] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7496.074933] Call Trace: [ 7496.074949] [ 7496.074955] ? show_regs.cold.9+0x22/0x2f [ 7496.074965] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7496.074972] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7496.074976] ? watchdog+0x40/0x40 [ 7496.074980] ? hrtimer_interrupt+0x128/0x2c0 [ 7496.074985] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7496.074993] ? apic_timer_interrupt+0xf/0x20 [ 7496.074997] [ 7496.075000] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7496.075004] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7496.075010] do_raw_spin_lock+0x177/0x190 [ 7496.075015] _raw_spin_lock+0x21/0x30 [ 7496.075019] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7496.075226] process_one_work+0x2c8/0x700 [ 7496.075233] worker_thread+0x296/0x6e0 [ 7496.075249] ? rescuer_thread+0x570/0x570 [ 7496.075253] kthread+0x1d1/0x200 [ 7496.075258] ? set_kthread_struct+0x70/0x70 [ 7496.075262] ret_from_fork+0x1f/0x30 [ 7496.270890] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7496.304075] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7496.311587] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7496.317767] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7496.321772] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7496.336955] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7496.342301] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7496.345421] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7496.348896] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7496.352163] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7496.355358] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7496.358702] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7496.362314] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7496.364897] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7496.368227] Call Trace: [ 7496.369305] [ 7496.370216] ? show_regs.cold.9+0x22/0x2f [ 7496.372297] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7496.374345] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7496.379475] ? watchdog+0x40/0x40 [ 7496.380785] ? hrtimer_interrupt+0x128/0x2c0 [ 7496.384756] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7496.386861] ? apic_timer_interrupt+0xf/0x20 [ 7496.389524] [ 7496.390495] ? native_safe_halt+0x2e/0x50 [ 7496.392527] kvm_wait+0xcd/0xe0 [ 7496.394170] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7496.398357] do_raw_spin_lock+0x177/0x190 [ 7496.400298] _raw_spin_lock+0x21/0x30 [ 7496.401904] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7496.405048] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7496.409167] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7496.413829] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7496.416490] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7496.420104] ? woken_wake_function+0x30/0x30 [ 7496.422483] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7496.428908] kthread+0x1d1/0x200 [ 7496.430078] ? set_kthread_struct+0x70/0x70 [ 7496.432758] ret_from_fork+0x1f/0x30 [ 7500.056493] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 7500.060602] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7500.091716] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7500.099376] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7500.103636] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7500.105818] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7500.118122] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7500.123597] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7500.128236] RDX: 0000000000006c51 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7500.132188] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7500.136738] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7500.139210] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7500.143081] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7500.150126] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7500.155967] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7500.159032] Call Trace: [ 7500.161255] [ 7500.162714] ? show_regs.cold.9+0x22/0x2f [ 7500.165395] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7500.169915] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7500.176762] ? watchdog+0x40/0x40 [ 7500.179531] ? hrtimer_interrupt+0x128/0x2c0 [ 7500.182303] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7500.185043] ? apic_timer_interrupt+0xf/0x20 [ 7500.187746] [ 7500.188737] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7500.192863] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7500.197854] do_raw_spin_lock+0x177/0x190 [ 7500.199842] _raw_spin_lock+0x21/0x30 [ 7500.201680] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7500.205633] obd_mod_free+0x46/0x140 [ptlrpc] [ 7500.210759] mdc_commit_open+0x91/0xb0 [mdc] [ 7500.214872] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7500.218611] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7500.222824] after_reply+0xe2c/0x1780 [ptlrpc] [ 7500.226292] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7500.230147] ? __next_timer_interrupt+0x160/0x160 [ 7500.235479] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7500.237557] ? do_wait_intr+0xf0/0xf0 [ 7500.239213] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7500.241305] kthread+0x1d1/0x200 [ 7500.242657] ? set_kthread_struct+0x70/0x70 [ 7500.244294] ret_from_fork+0x1f/0x30 [ 7505.888194] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4992s! [ 7505.893770] Showing busy workqueues and worker pools: [ 7505.898556] workqueue events: flags=0x0 [ 7505.903079] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7505.908909] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7505.919775] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7505.924107] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7505.930636] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7505.936573] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7505.944200] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7505.950381] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7505.959084] workqueue events_long: flags=0x0 [ 7505.964785] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7505.972550] pending: lu_objects_flush [obdclass] [ 7505.975501] workqueue events_unbound: flags=0x2 [ 7505.980401] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7505.985522] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7505.988962] workqueue events_power_efficient: flags=0x80 [ 7505.994795] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7506.000197] pending: neigh_periodic_work, do_cache_clean [ 7506.005868] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7506.012119] pending: neigh_periodic_work, check_lifetime [ 7506.016560] workqueue mm_percpu_wq: flags=0x8 [ 7506.020198] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7506.025056] pending: vmstat_update, lru_add_drain_per_cpu [ 7506.029567] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7506.036113] pending: vmstat_update [ 7506.040521] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7506.047644] pending: lru_add_drain_per_cpu BAR(48) [ 7506.055279] workqueue ipv6_addrconf: flags=0x40008 [ 7506.058132] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7506.071650] pending: addrconf_verify_work [ 7506.081276] workqueue knbd0-recv: flags=0x1a [ 7506.082817] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7506.091722] in-flight: 650:recv_work [ 7506.094635] workqueue ptlrpc_pinger: flags=0x2 [ 7506.099040] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7506.103606] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7506.110735] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4992s workers=4 idle: 36 466 29757 [ 7506.118880] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7496s workers=2 idle: 652 [ 7520.062050] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7520.066788] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7520.103490] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7520.109161] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7520.119809] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7520.124643] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7520.139387] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7520.151085] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7520.162545] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7520.166696] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7520.170848] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7520.174701] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7520.180367] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7520.190370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7520.197102] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7520.203546] Call Trace: [ 7520.204656] [ 7520.205615] ? show_regs.cold.9+0x22/0x2f [ 7520.206954] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7520.209463] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7520.211883] ? watchdog+0x40/0x40 [ 7520.216827] ? hrtimer_interrupt+0x128/0x2c0 [ 7520.220884] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7520.223215] ? apic_timer_interrupt+0xf/0x20 [ 7520.227778] [ 7520.228973] ? native_safe_halt+0x2e/0x50 [ 7520.235039] kvm_wait+0xcd/0xe0 [ 7520.236380] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7520.238600] do_raw_spin_lock+0x177/0x190 [ 7520.243683] _raw_spin_lock+0x21/0x30 [ 7520.249427] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7520.252203] ? do_raw_spin_unlock+0x75/0x190 [ 7520.254213] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7520.257651] ? do_wait_intr+0xf0/0xf0 [ 7520.259664] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7520.264083] mdc_reint+0x2d/0x110 [mdc] [ 7520.265890] mdc_unlink+0x658/0x870 [mdc] [ 7520.268046] lmv_unlink+0x34d/0xbd0 [lmv] [ 7520.270623] ll_unlink+0x24d/0x7f0 [lustre] [ 7520.273552] vfs_unlink+0x202/0x330 [ 7520.274684] do_unlinkat+0x2fa/0x460 [ 7520.275771] __x64_sys_unlink+0x37/0x50 [ 7520.277169] do_syscall_64+0xc1/0x440 [ 7520.278875] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7520.281875] RIP: 0033:0x7f83e5f5c10b [ 7520.283377] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7520.293116] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7520.296652] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7520.299976] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7520.302425] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7520.305370] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7520.307738] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7524.059017] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 7524.063101] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7524.063465] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 7524.099300] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7524.099313] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7524.099316] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7524.099328] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7524.099332] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7524.099337] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7524.099339] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7524.099341] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7524.099343] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7524.099345] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7524.099348] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7524.099350] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7524.099352] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7524.099360] Call Trace: [ 7524.099365] [ 7524.099370] ? show_regs.cold.9+0x22/0x2f [ 7524.099379] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7524.099385] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7524.099389] ? watchdog+0x40/0x40 [ 7524.099393] ? hrtimer_interrupt+0x128/0x2c0 [ 7524.099397] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7524.099402] ? apic_timer_interrupt+0xf/0x20 [ 7524.099407] [ 7524.099410] ? native_safe_halt+0x2e/0x50 [ 7524.103865] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7524.103945] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7524.103952] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7524.103958] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7524.110802] kvm_wait+0xcd/0xe0 [ 7524.110824] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7524.110835] do_raw_spin_lock+0x177/0x190 [ 7524.110840] _raw_spin_lock+0x21/0x30 [ 7524.110848] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7524.111137] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7524.112035] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7524.112539] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7524.112758] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7524.117875] [ 7524.117884] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7524.117902] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7524.117907] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7524.117912] RAX: 0000000000006a88 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7524.117914] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7524.117916] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7524.117918] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7524.117921] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7524.117923] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7524.117927] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7524.117929] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7524.117938] Call Trace: [ 7524.117942] [ 7524.117948] ? show_regs.cold.9+0x22/0x2f [ 7524.117957] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7524.117974] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7524.117979] ? watchdog+0x40/0x40 [ 7524.117985] ? hrtimer_interrupt+0x128/0x2c0 [ 7524.117989] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7524.117998] ? apic_timer_interrupt+0xf/0x20 [ 7524.118003] [ 7524.118006] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7524.118011] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7524.118017] do_raw_spin_lock+0x177/0x190 [ 7524.118023] _raw_spin_lock+0x21/0x30 [ 7524.118028] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7524.118541] process_one_work+0x2c8/0x700 [ 7524.118548] worker_thread+0x296/0x6e0 [ 7524.118553] ? rescuer_thread+0x570/0x570 [ 7524.118557] kthread+0x1d1/0x200 [ 7524.118561] ? set_kthread_struct+0x70/0x70 [ 7524.118565] ret_from_fork+0x1f/0x30 [ 7524.507501] ? woken_wake_function+0x30/0x30 [ 7524.512273] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7524.516974] kthread+0x1d1/0x200 [ 7524.519702] ? set_kthread_struct+0x70/0x70 [ 7524.522895] ret_from_fork+0x1f/0x30 [ 7532.059560] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 7532.068842] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7532.102408] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7532.114875] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7532.120073] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7532.122558] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7532.133293] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7532.137397] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7532.142527] RDX: 00000000000069e5 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7532.148179] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7532.151529] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7532.156425] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7532.160308] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7532.166803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7532.171071] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7532.180045] Call Trace: [ 7532.184178] [ 7532.186656] ? show_regs.cold.9+0x22/0x2f [ 7532.188710] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7532.191326] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7532.193893] ? watchdog+0x40/0x40 [ 7532.197169] ? hrtimer_interrupt+0x128/0x2c0 [ 7532.199725] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7532.202779] ? apic_timer_interrupt+0xf/0x20 [ 7532.205873] [ 7532.206903] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7532.209527] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7532.212249] do_raw_spin_lock+0x177/0x190 [ 7532.213902] _raw_spin_lock+0x21/0x30 [ 7532.216279] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7532.223349] obd_mod_free+0x46/0x140 [ptlrpc] [ 7532.229293] mdc_commit_open+0x91/0xb0 [mdc] [ 7532.234797] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7532.241533] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7532.245990] after_reply+0xe2c/0x1780 [ptlrpc] [ 7532.249801] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7532.253050] ? __next_timer_interrupt+0x160/0x160 [ 7532.255805] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7532.282012] ? do_wait_intr+0xf0/0xf0 [ 7532.283412] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7532.292786] kthread+0x1d1/0x200 [ 7532.295474] ? set_kthread_struct+0x70/0x70 [ 7532.310997] ret_from_fork+0x1f/0x30 [ 7536.620427] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5023s! [ 7536.628418] Showing busy workqueues and worker pools: [ 7536.646718] workqueue events: flags=0x0 [ 7536.659943] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7536.677569] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7536.686942] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7536.709178] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7536.729488] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7536.748275] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7536.761227] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7536.782456] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7536.813467] workqueue events_long: flags=0x0 [ 7536.823829] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7536.847049] pending: lu_objects_flush [obdclass] [ 7536.879153] workqueue events_unbound: flags=0x2 [ 7536.890926] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7536.906640] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7536.925096] workqueue events_power_efficient: flags=0x80 [ 7536.953922] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7536.984204] pending: neigh_periodic_work, do_cache_clean [ 7537.007347] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7537.046545] pending: neigh_periodic_work, check_lifetime [ 7537.048531] workqueue mm_percpu_wq: flags=0x8 [ 7537.063300] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7537.081878] pending: vmstat_update, lru_add_drain_per_cpu [ 7537.113695] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7537.126215] pending: vmstat_update [ 7537.145406] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7537.171018] pending: lru_add_drain_per_cpu BAR(48) [ 7537.183601] workqueue ipv6_addrconf: flags=0x40008 [ 7537.196302] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7537.206992] pending: addrconf_verify_work [ 7537.214188] workqueue knbd0-recv: flags=0x1a [ 7537.228384] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7537.240765] in-flight: 650:recv_work [ 7537.249445] workqueue ptlrpc_pinger: flags=0x2 [ 7537.258364] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7537.271401] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7537.286316] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5024s workers=4 idle: 36 466 29757 [ 7537.303196] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7527s workers=2 idle: 652 [ 7548.062035] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7548.065244] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7548.085577] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7548.089591] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7548.095229] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7548.098558] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7548.110246] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7548.122350] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7548.127133] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7548.133329] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7548.137477] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7548.141109] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7548.144987] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7548.149726] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7548.153611] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7548.158579] Call Trace: [ 7548.159862] [ 7548.161176] ? show_regs.cold.9+0x22/0x2f [ 7548.163489] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7548.166508] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7548.168479] ? watchdog+0x40/0x40 [ 7548.170176] ? hrtimer_interrupt+0x128/0x2c0 [ 7548.172027] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7548.174336] ? apic_timer_interrupt+0xf/0x20 [ 7548.176542] [ 7548.177662] ? native_safe_halt+0x2e/0x50 [ 7548.179370] kvm_wait+0xcd/0xe0 [ 7548.180703] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7548.183160] do_raw_spin_lock+0x177/0x190 [ 7548.184362] _raw_spin_lock+0x21/0x30 [ 7548.185766] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7548.188945] ? do_raw_spin_unlock+0x75/0x190 [ 7548.191136] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7548.193628] ? do_wait_intr+0xf0/0xf0 [ 7548.195875] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7548.198172] mdc_reint+0x2d/0x110 [mdc] [ 7548.199634] mdc_unlink+0x658/0x870 [mdc] [ 7548.201395] lmv_unlink+0x34d/0xbd0 [lmv] [ 7548.203289] ll_unlink+0x24d/0x7f0 [lustre] [ 7548.205284] vfs_unlink+0x202/0x330 [ 7548.206682] do_unlinkat+0x2fa/0x460 [ 7548.208437] __x64_sys_unlink+0x37/0x50 [ 7548.209853] do_syscall_64+0xc1/0x440 [ 7548.211622] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7548.213693] RIP: 0033:0x7f83e5f5c10b [ 7548.215503] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7548.223536] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7548.227103] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7548.230612] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7548.233393] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7548.236375] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7548.239583] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7552.059024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 7552.060018] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 7552.062582] Modules linked in: [ 7552.074290] Modules linked in: [ 7552.074302] lustre(O) [ 7552.075328] lustre(O) [ 7552.077777] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7552.077863] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7552.077873] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7552.077877] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7552.083514] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7552.083613] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7552.083623] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7552.083631] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7552.086447] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7552.086459] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7552.086466] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7552.086469] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7552.086472] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7552.086474] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7552.086477] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7552.086480] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7552.086482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7552.086485] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7552.086493] Call Trace: [ 7552.086500] [ 7552.086507] ? show_regs.cold.9+0x22/0x2f [ 7552.086523] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7552.086532] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7552.086536] ? watchdog+0x40/0x40 [ 7552.086541] ? hrtimer_interrupt+0x128/0x2c0 [ 7552.086546] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7552.086556] ? apic_timer_interrupt+0xf/0x20 [ 7552.086561] [ 7552.086564] ? native_safe_halt+0x2e/0x50 [ 7552.086568] kvm_wait+0xcd/0xe0 [ 7552.086578] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7552.086588] do_raw_spin_lock+0x177/0x190 [ 7552.086593] _raw_spin_lock+0x21/0x30 [ 7552.086598] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7552.114457] [ 7552.114469] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7552.114491] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7552.114494] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7552.114498] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7552.114500] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7552.114502] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7552.114504] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7552.114506] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7552.114509] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7552.114512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7552.114513] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7552.114519] Call Trace: [ 7552.114525] [ 7552.114531] ? show_regs.cold.9+0x22/0x2f [ 7552.114540] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7552.114545] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7552.114549] ? watchdog+0x40/0x40 [ 7552.114552] ? hrtimer_interrupt+0x128/0x2c0 [ 7552.114554] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7552.114560] ? apic_timer_interrupt+0xf/0x20 [ 7552.114563] [ 7552.114565] ? native_safe_halt+0x2e/0x50 [ 7552.114568] kvm_wait+0xcd/0xe0 [ 7552.114576] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7552.114585] do_raw_spin_lock+0x177/0x190 [ 7552.114589] _raw_spin_lock+0x21/0x30 [ 7552.114592] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7552.114965] process_one_work+0x2c8/0x700 [ 7552.114974] worker_thread+0x296/0x6e0 [ 7552.114979] ? rescuer_thread+0x570/0x570 [ 7552.114983] kthread+0x1d1/0x200 [ 7552.114989] ? set_kthread_struct+0x70/0x70 [ 7552.114993] ret_from_fork+0x1f/0x30 [ 7552.438388] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7552.443192] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7552.447670] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7552.450765] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7552.454014] ? woken_wake_function+0x30/0x30 [ 7552.456025] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7552.463951] kthread+0x1d1/0x200 [ 7552.465840] ? set_kthread_struct+0x70/0x70 [ 7552.467789] ret_from_fork+0x1f/0x30 [ 7564.056033] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 7564.059318] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7564.078495] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7564.083243] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7564.086516] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7564.088590] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7564.094970] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7564.097370] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7564.099488] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7564.101570] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7564.103698] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7564.106005] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7564.109143] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7564.111787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7564.113874] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7564.116127] Call Trace: [ 7564.117358] [ 7564.118088] ? show_regs.cold.9+0x22/0x2f [ 7564.119455] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7564.121337] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7564.122691] ? watchdog+0x40/0x40 [ 7564.124197] ? hrtimer_interrupt+0x128/0x2c0 [ 7564.125588] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7564.127661] ? apic_timer_interrupt+0xf/0x20 [ 7564.129237] [ 7564.129739] ? native_safe_halt+0x2e/0x50 [ 7564.131282] kvm_wait+0xcd/0xe0 [ 7564.132187] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7564.133729] do_raw_spin_lock+0x177/0x190 [ 7564.135190] _raw_spin_lock+0x21/0x30 [ 7564.137222] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7564.139593] obd_mod_free+0x46/0x140 [ptlrpc] [ 7564.141590] mdc_commit_open+0x91/0xb0 [mdc] [ 7564.143115] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7564.145451] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7564.147690] after_reply+0xe2c/0x1780 [ptlrpc] [ 7564.149660] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7564.152144] ? __next_timer_interrupt+0x160/0x160 [ 7564.153497] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7564.155218] ? do_wait_intr+0xf0/0xf0 [ 7564.156335] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7564.158262] kthread+0x1d1/0x200 [ 7564.159321] ? set_kthread_struct+0x70/0x70 [ 7564.160737] ret_from_fork+0x1f/0x30 [ 7567.329084] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5054s! [ 7567.335060] Showing busy workqueues and worker pools: [ 7567.339856] workqueue events: flags=0x0 [ 7567.343296] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7567.361512] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7567.379123] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7567.382770] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7567.388360] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7567.397371] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7567.402013] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7567.413566] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7567.420190] workqueue events_long: flags=0x0 [ 7567.421876] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7567.427433] pending: lu_objects_flush [obdclass] [ 7567.429454] workqueue events_unbound: flags=0x2 [ 7567.434424] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7567.441148] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7567.444922] workqueue events_power_efficient: flags=0x80 [ 7567.448947] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7567.453982] pending: neigh_periodic_work, do_cache_clean [ 7567.460158] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7567.465069] pending: neigh_periodic_work, check_lifetime [ 7567.468171] workqueue mm_percpu_wq: flags=0x8 [ 7567.474068] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7567.479696] pending: vmstat_update, lru_add_drain_per_cpu [ 7567.482718] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7567.489346] pending: vmstat_update [ 7567.490960] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7567.497735] pending: lru_add_drain_per_cpu BAR(48) [ 7567.503470] workqueue ipv6_addrconf: flags=0x40008 [ 7567.509832] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7567.514758] pending: addrconf_verify_work [ 7567.518924] workqueue knbd0-recv: flags=0x1a [ 7567.523148] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7567.532438] in-flight: 650:recv_work [ 7567.534408] workqueue ptlrpc_pinger: flags=0x2 [ 7567.538424] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7567.544929] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7567.553177] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5054s workers=4 idle: 36 466 29757 [ 7567.560050] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7557s workers=2 idle: 652 [ 7576.062513] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7576.066606] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7576.088005] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7576.092249] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7576.095321] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7576.098070] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7576.104798] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7576.107336] RAX: 0000000000003e95 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7576.109669] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7576.112295] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7576.114902] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7576.117152] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7576.119196] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7576.122072] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7576.123999] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7576.126239] Call Trace: [ 7576.127170] [ 7576.127752] ? show_regs.cold.9+0x22/0x2f [ 7576.129259] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7576.130961] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7576.132670] ? watchdog+0x40/0x40 [ 7576.134324] ? hrtimer_interrupt+0x128/0x2c0 [ 7576.135964] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7576.138110] ? apic_timer_interrupt+0xf/0x20 [ 7576.139342] [ 7576.140093] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7576.142147] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7576.144032] do_raw_spin_lock+0x177/0x190 [ 7576.146039] _raw_spin_lock+0x21/0x30 [ 7576.147104] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7576.149062] ? do_raw_spin_unlock+0x75/0x190 [ 7576.150429] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7576.152789] ? do_wait_intr+0xf0/0xf0 [ 7576.154255] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7576.156687] mdc_reint+0x2d/0x110 [mdc] [ 7576.158387] mdc_unlink+0x658/0x870 [mdc] [ 7576.159905] lmv_unlink+0x34d/0xbd0 [lmv] [ 7576.161542] ll_unlink+0x24d/0x7f0 [lustre] [ 7576.163139] vfs_unlink+0x202/0x330 [ 7576.164449] do_unlinkat+0x2fa/0x460 [ 7576.165865] __x64_sys_unlink+0x37/0x50 [ 7576.167424] do_syscall_64+0xc1/0x440 [ 7576.168868] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7576.171625] RIP: 0033:0x7f83e5f5c10b [ 7576.172845] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7576.180757] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7576.183952] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7576.186398] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7576.189042] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7576.191530] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7576.194375] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7580.060078] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 7580.065694] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 7580.081609] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7580.081760] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7580.081767] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7580.081774] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7580.088793] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7580.088966] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7580.088976] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7580.088979] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7580.088997] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7580.089003] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7580.089008] RAX: 0000000000007747 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7580.089011] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7580.089014] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7580.089019] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7580.089023] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7580.089028] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7580.089032] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7580.089036] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7580.089044] Call Trace: [ 7580.089051] [ 7580.089059] ? show_regs.cold.9+0x22/0x2f [ 7580.089071] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7580.089080] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7580.089085] ? watchdog+0x40/0x40 [ 7580.089090] ? hrtimer_interrupt+0x128/0x2c0 [ 7580.089095] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7580.089105] ? apic_timer_interrupt+0xf/0x20 [ 7580.089111] [ 7580.089114] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7580.089119] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7580.089126] do_raw_spin_lock+0x177/0x190 [ 7580.089131] _raw_spin_lock+0x21/0x30 [ 7580.089136] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7580.089608] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7580.090479] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7580.090847] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7580.091137] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7580.091493] ? woken_wake_function+0x30/0x30 [ 7580.091506] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7580.120980] [ 7580.121000] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7580.121026] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7580.121033] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7580.121045] RAX: 0000000000002499 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7580.121049] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7580.121051] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7580.121054] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7580.121057] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7580.121061] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7580.121064] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7580.121067] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7580.121081] Call Trace: [ 7580.121089] [ 7580.121096] ? show_regs.cold.9+0x22/0x2f [ 7580.121113] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7580.121121] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7580.121126] ? watchdog+0x40/0x40 [ 7580.121131] ? hrtimer_interrupt+0x128/0x2c0 [ 7580.121135] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7580.121149] ? apic_timer_interrupt+0xf/0x20 [ 7580.121154] [ 7580.121157] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7580.121162] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7580.121174] do_raw_spin_lock+0x177/0x190 [ 7580.121179] _raw_spin_lock+0x21/0x30 [ 7580.121185] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7580.121870] process_one_work+0x2c8/0x700 [ 7580.121879] worker_thread+0x296/0x6e0 [ 7580.121884] ? rescuer_thread+0x570/0x570 [ 7580.121888] kthread+0x1d1/0x200 [ 7580.121894] ? set_kthread_struct+0x70/0x70 [ 7580.121897] ret_from_fork+0x1f/0x30 [ 7580.345931] kthread+0x1d1/0x200 [ 7580.347450] ? set_kthread_struct+0x70/0x70 [ 7580.349686] ret_from_fork+0x1f/0x30 [ 7592.056028] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 7592.059482] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7592.083289] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7592.088990] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7592.093549] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7592.095756] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7592.104694] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7592.108290] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7592.111830] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7592.115567] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7592.118889] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7592.121857] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7592.125192] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7592.128425] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7592.130618] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7592.133951] Call Trace: [ 7592.135094] [ 7592.136102] ? show_regs.cold.9+0x22/0x2f [ 7592.138106] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7592.140252] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7592.142483] ? watchdog+0x40/0x40 [ 7592.144051] ? hrtimer_interrupt+0x128/0x2c0 [ 7592.146854] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7592.149245] ? apic_timer_interrupt+0xf/0x20 [ 7592.150810] [ 7592.151522] ? native_safe_halt+0x2e/0x50 [ 7592.152997] kvm_wait+0xcd/0xe0 [ 7592.154890] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7592.157617] do_raw_spin_lock+0x177/0x190 [ 7592.159363] _raw_spin_lock+0x21/0x30 [ 7592.160950] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7592.163581] obd_mod_free+0x46/0x140 [ptlrpc] [ 7592.165698] mdc_commit_open+0x91/0xb0 [mdc] [ 7592.167721] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7592.170285] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7592.172885] after_reply+0xe2c/0x1780 [ptlrpc] [ 7592.176417] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7592.179081] ? __next_timer_interrupt+0x160/0x160 [ 7592.181078] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7592.182992] ? do_wait_intr+0xf0/0xf0 [ 7592.184875] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7592.187582] kthread+0x1d1/0x200 [ 7592.189160] ? set_kthread_struct+0x70/0x70 [ 7592.190900] ret_from_fork+0x1f/0x30 [ 7598.048045] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5084s! [ 7598.051039] Showing busy workqueues and worker pools: [ 7598.052973] workqueue events: flags=0x0 [ 7598.054734] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7598.057402] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7598.061070] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7598.063842] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7598.066415] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7598.068830] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7598.072025] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7598.074724] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7598.078328] workqueue events_long: flags=0x0 [ 7598.080034] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7598.082782] pending: lu_objects_flush [obdclass] [ 7598.084981] workqueue events_unbound: flags=0x2 [ 7598.090799] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7598.093397] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7598.096544] workqueue events_power_efficient: flags=0x80 [ 7598.098950] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7598.102416] pending: neigh_periodic_work, do_cache_clean [ 7598.105138] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7598.108173] pending: neigh_periodic_work, check_lifetime [ 7598.110974] workqueue mm_percpu_wq: flags=0x8 [ 7598.113670] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7598.116664] pending: vmstat_update, lru_add_drain_per_cpu [ 7598.119338] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7598.122087] pending: vmstat_update [ 7598.123623] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7598.126207] pending: lru_add_drain_per_cpu BAR(48) [ 7598.128352] workqueue ipv6_addrconf: flags=0x40008 [ 7598.130322] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7598.132858] pending: addrconf_verify_work [ 7598.134731] workqueue knbd0-recv: flags=0x1a [ 7598.137839] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7598.148114] in-flight: 650:recv_work [ 7598.151152] workqueue ptlrpc_pinger: flags=0x2 [ 7598.155976] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7598.159419] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7598.162970] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5084s workers=4 idle: 36 466 29757 [ 7598.166661] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7588s workers=2 idle: 652 [ 7604.067499] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7604.073391] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7604.104284] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7604.116005] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7604.121698] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7604.125026] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7604.138168] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7604.147971] RAX: 000000000000730b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7604.151538] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7604.160944] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7604.165411] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7604.172553] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7604.182091] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7604.189203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7604.198075] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7604.205123] Call Trace: [ 7604.208280] [ 7604.209041] ? show_regs.cold.9+0x22/0x2f [ 7604.213987] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7604.216212] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7604.221057] ? watchdog+0x40/0x40 [ 7604.223371] ? hrtimer_interrupt+0x128/0x2c0 [ 7604.226432] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7604.229019] ? apic_timer_interrupt+0xf/0x20 [ 7604.232309] [ 7604.233901] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7604.240902] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7604.243537] do_raw_spin_lock+0x177/0x190 [ 7604.245294] _raw_spin_lock+0x21/0x30 [ 7604.247175] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7604.249856] ? do_raw_spin_unlock+0x75/0x190 [ 7604.251786] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7604.254795] ? do_wait_intr+0xf0/0xf0 [ 7604.256465] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7604.259290] mdc_reint+0x2d/0x110 [mdc] [ 7604.261045] mdc_unlink+0x658/0x870 [mdc] [ 7604.262996] lmv_unlink+0x34d/0xbd0 [lmv] [ 7604.264929] ll_unlink+0x24d/0x7f0 [lustre] [ 7604.266621] vfs_unlink+0x202/0x330 [ 7604.267789] do_unlinkat+0x2fa/0x460 [ 7604.269622] __x64_sys_unlink+0x37/0x50 [ 7604.271290] do_syscall_64+0xc1/0x440 [ 7604.273099] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7604.274810] RIP: 0033:0x7f83e5f5c10b [ 7604.275987] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7604.283082] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7604.285992] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7604.288699] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7604.291473] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7604.294224] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7604.297011] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7608.059033] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 7608.064359] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) [ 7608.065459] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 7608.065988] lfsck(O) [ 7608.071615] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7608.071694] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7608.071701] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7608.071707] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7608.071991] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7608.072002] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7608.072006] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7608.072011] RAX: 0000000000007a06 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7608.072014] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7608.072015] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7608.072017] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7608.072020] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7608.072023] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7608.072025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7608.072028] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7608.072038] Call Trace: [ 7608.072043] [ 7608.072049] ? show_regs.cold.9+0x22/0x2f [ 7608.072059] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7608.072066] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7608.072070] ? watchdog+0x40/0x40 [ 7608.072074] ? hrtimer_interrupt+0x128/0x2c0 [ 7608.072078] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7608.072086] ? apic_timer_interrupt+0xf/0x20 [ 7608.074606] [ 7608.074614] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7608.074627] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7608.074634] do_raw_spin_lock+0x177/0x190 [ 7608.074640] _raw_spin_lock+0x21/0x30 [ 7608.074647] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7608.080965] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7608.081029] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7608.081037] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7608.081040] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7608.081053] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7608.081058] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7608.081062] RAX: 0000000000003d90 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7608.081065] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7608.081067] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7608.081069] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7608.081071] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7608.081074] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7608.081077] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7608.081079] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7608.081089] Call Trace: [ 7608.081094] [ 7608.081099] ? show_regs.cold.9+0x22/0x2f [ 7608.081108] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7608.081114] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7608.081118] ? watchdog+0x40/0x40 [ 7608.081122] ? hrtimer_interrupt+0x128/0x2c0 [ 7608.081126] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7608.081133] ? apic_timer_interrupt+0xf/0x20 [ 7608.081138] [ 7608.081141] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7608.081146] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7608.135058] process_one_work+0x2c8/0x700 [ 7608.135078] worker_thread+0x296/0x6e0 [ 7608.135082] ? rescuer_thread+0x570/0x570 [ 7608.135086] kthread+0x1d1/0x200 [ 7608.135912] ? set_kthread_struct+0x70/0x70 [ 7608.135920] ret_from_fork+0x1f/0x30 [ 7608.413604] do_raw_spin_lock+0x177/0x190 [ 7608.416254] _raw_spin_lock+0x21/0x30 [ 7608.417773] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7608.421441] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7608.424224] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7608.426958] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7608.429845] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7608.432575] ? woken_wake_function+0x30/0x30 [ 7608.434319] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7608.441673] kthread+0x1d1/0x200 [ 7608.444208] ? set_kthread_struct+0x70/0x70 [ 7608.449954] ret_from_fork+0x1f/0x30 [ 7621.093843] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 7621.097732] rcu: 0-....: (3598741 ticks this GP) idle=30a/1/0x4000000000000000 softirq=128519/128519 fqs=126395 [ 7621.107891] (detected by 1, t=5640134 jiffies, g=262441, q=58806) [ 7621.113217] Sending NMI from CPU 1 to CPUs 0: [ 7621.116051] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 7621.121485] rcu: rcu_sched kthread starved for 5109321 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 7621.132899] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7621.142702] rcu: RCU grace-period kthread stack dump: [ 7621.148487] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7621.153575] Call Trace: [ 7621.154562] __schedule+0x351/0xcb0 [ 7621.156465] schedule+0xc0/0x180 [ 7621.157755] schedule_timeout+0xb4/0x190 [ 7621.159844] ? do_raw_spin_unlock+0x75/0x190 [ 7621.161381] ? __next_timer_interrupt+0x160/0x160 [ 7621.165796] ? prepare_to_swait_event+0xf3/0x140 [ 7621.169528] rcu_gp_kthread+0x8dd/0x15b0 [ 7621.170804] ? force_qs_rnp+0x2c0/0x2c0 [ 7621.172405] kthread+0x1d1/0x200 [ 7621.173760] ? set_kthread_struct+0x70/0x70 [ 7621.177732] ret_from_fork+0x1f/0x30 [ 7621.181527] rcu: Stack dump where RCU GP kthread last ran: [ 7621.184452] Sending NMI from CPU 1 to CPUs 0: [ 7621.188758] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 7628.256273] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5115s! [ 7628.259769] Showing busy workqueues and worker pools: [ 7628.266458] workqueue events: flags=0x0 [ 7628.270088] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7628.280095] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7628.294441] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7628.300497] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7628.316299] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7628.327070] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7628.338445] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7628.348975] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7628.360575] workqueue events_long: flags=0x0 [ 7628.366341] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7628.372008] pending: lu_objects_flush [obdclass] [ 7628.381231] workqueue events_unbound: flags=0x2 [ 7628.389756] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7628.395239] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7628.407519] workqueue events_power_efficient: flags=0x80 [ 7628.415859] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7628.421930] pending: neigh_periodic_work, do_cache_clean [ 7628.428470] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7628.436157] pending: neigh_periodic_work, check_lifetime [ 7628.445460] workqueue mm_percpu_wq: flags=0x8 [ 7628.450715] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7628.462256] pending: vmstat_update, lru_add_drain_per_cpu [ 7628.465512] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7628.473153] pending: vmstat_update [ 7628.478733] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7628.482801] pending: lru_add_drain_per_cpu BAR(48) [ 7628.485511] workqueue ipv6_addrconf: flags=0x40008 [ 7628.488776] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7628.493691] pending: addrconf_verify_work [ 7628.498948] workqueue knbd0-recv: flags=0x1a [ 7628.502065] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7628.509128] in-flight: 650:recv_work [ 7628.512769] workqueue ptlrpc_pinger: flags=0x2 [ 7628.515399] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7628.521509] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7628.526961] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5115s workers=4 idle: 36 466 29757 [ 7628.532546] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7618s workers=2 idle: 652 [ 7632.062137] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7632.065222] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7632.105366] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7632.113472] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7632.122004] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7632.125620] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7632.137579] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7632.142892] RAX: 0000000000006d7e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7632.153588] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7632.165356] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7632.169898] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7632.182476] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7632.188948] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7632.194831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7632.207656] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7632.215979] Call Trace: [ 7632.217432] [ 7632.218470] ? show_regs.cold.9+0x22/0x2f [ 7632.221766] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7632.227699] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7632.230228] ? watchdog+0x40/0x40 [ 7632.232303] ? hrtimer_interrupt+0x128/0x2c0 [ 7632.234637] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7632.239221] ? apic_timer_interrupt+0xf/0x20 [ 7632.245041] [ 7632.246343] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7632.250711] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7632.253994] do_raw_spin_lock+0x177/0x190 [ 7632.259411] _raw_spin_lock+0x21/0x30 [ 7632.265682] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7632.268740] ? do_raw_spin_unlock+0x75/0x190 [ 7632.271432] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7632.274541] ? do_wait_intr+0xf0/0xf0 [ 7632.276434] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7632.279261] mdc_reint+0x2d/0x110 [mdc] [ 7632.281087] mdc_unlink+0x658/0x870 [mdc] [ 7632.283060] lmv_unlink+0x34d/0xbd0 [lmv] [ 7632.285340] ll_unlink+0x24d/0x7f0 [lustre] [ 7632.287419] vfs_unlink+0x202/0x330 [ 7632.288931] do_unlinkat+0x2fa/0x460 [ 7632.296974] __x64_sys_unlink+0x37/0x50 [ 7632.306586] do_syscall_64+0xc1/0x440 [ 7632.310073] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7632.313068] RIP: 0033:0x7f83e5f5c10b [ 7632.320110] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7632.333737] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7632.337674] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7632.344524] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7632.348104] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7632.355577] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7632.366127] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7636.061074] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 7636.068134] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7636.102781] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7636.114589] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7636.118568] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7636.123829] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7636.130015] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7636.143969] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7636.147753] RAX: 000000000000578d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7636.151780] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7636.156682] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7636.162310] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7636.169826] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7636.173755] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7636.176874] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7636.179420] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7636.182286] Call Trace: [ 7636.183827] [ 7636.184717] ? show_regs.cold.9+0x22/0x2f [ 7636.189137] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7636.191310] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7636.193582] ? watchdog+0x40/0x40 [ 7636.195037] ? hrtimer_interrupt+0x128/0x2c0 [ 7636.197222] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7636.199176] ? apic_timer_interrupt+0xf/0x20 [ 7636.203749] [ 7636.204381] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7636.205820] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7636.209251] do_raw_spin_lock+0x177/0x190 [ 7636.210609] _raw_spin_lock+0x21/0x30 [ 7636.212253] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7636.214457] process_one_work+0x2c8/0x700 [ 7636.216044] worker_thread+0x296/0x6e0 [ 7636.217765] ? rescuer_thread+0x570/0x570 [ 7636.219431] kthread+0x1d1/0x200 [ 7636.220858] ? set_kthread_struct+0x70/0x70 [ 7636.222621] ret_from_fork+0x1f/0x30 [ 7648.059758] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 7648.063333] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7648.092520] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7648.102797] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7648.112585] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7648.116597] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7648.133814] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7648.138608] RAX: 0000000000007d8d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7648.142321] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7648.148694] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7648.154835] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7648.162202] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7648.165431] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7648.168607] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7648.172154] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7648.176945] Call Trace: [ 7648.178110] [ 7648.180781] ? show_regs.cold.9+0x22/0x2f [ 7648.183800] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7648.189510] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7648.193680] ? watchdog+0x40/0x40 [ 7648.197523] ? hrtimer_interrupt+0x128/0x2c0 [ 7648.201211] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7648.204959] ? apic_timer_interrupt+0xf/0x20 [ 7648.210300] [ 7648.211236] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7648.217284] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7648.219275] do_raw_spin_lock+0x177/0x190 [ 7648.224691] _raw_spin_lock+0x21/0x30 [ 7648.227546] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7648.232082] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7648.234661] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7648.238509] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7648.249330] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7648.251495] ? woken_wake_function+0x30/0x30 [ 7648.255931] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7648.260749] kthread+0x1d1/0x200 [ 7648.262626] ? set_kthread_struct+0x70/0x70 [ 7648.266773] ret_from_fork+0x1f/0x30 [ 7656.058733] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 7656.061196] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7656.081563] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7656.086925] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7656.090056] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7656.091977] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7656.100379] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7656.104097] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7656.107336] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7656.110384] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7656.113758] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7656.116961] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7656.119821] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7656.123478] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7656.125660] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7656.129030] Call Trace: [ 7656.130019] [ 7656.130768] ? show_regs.cold.9+0x22/0x2f [ 7656.132298] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7656.134581] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7656.136369] ? watchdog+0x40/0x40 [ 7656.138262] ? hrtimer_interrupt+0x128/0x2c0 [ 7656.139770] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7656.141735] ? apic_timer_interrupt+0xf/0x20 [ 7656.143674] [ 7656.144549] ? native_safe_halt+0x2e/0x50 [ 7656.146397] kvm_wait+0xcd/0xe0 [ 7656.147550] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7656.149975] do_raw_spin_lock+0x177/0x190 [ 7656.151439] _raw_spin_lock+0x21/0x30 [ 7656.153195] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7656.155161] obd_mod_free+0x46/0x140 [ptlrpc] [ 7656.156990] mdc_commit_open+0x91/0xb0 [mdc] [ 7656.158599] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7656.161060] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7656.163985] after_reply+0xe2c/0x1780 [ptlrpc] [ 7656.166000] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7656.168169] ? __next_timer_interrupt+0x160/0x160 [ 7656.169986] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7656.171856] ? do_wait_intr+0xf0/0xf0 [ 7656.173388] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7656.175886] kthread+0x1d1/0x200 [ 7656.177033] ? set_kthread_struct+0x70/0x70 [ 7656.179079] ret_from_fork+0x1f/0x30 [ 7658.976065] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5145s! [ 7658.980989] Showing busy workqueues and worker pools: [ 7658.984080] workqueue events: flags=0x0 [ 7658.986836] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7658.994272] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7659.008310] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7659.012194] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7659.015475] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7659.022046] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7659.027247] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7659.032284] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7659.039184] workqueue events_long: flags=0x0 [ 7659.043275] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7659.049556] pending: lu_objects_flush [obdclass] [ 7659.053635] workqueue events_unbound: flags=0x2 [ 7659.056993] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7659.061118] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7659.067848] workqueue events_power_efficient: flags=0x80 [ 7659.080492] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7659.085275] pending: neigh_periodic_work, do_cache_clean [ 7659.096190] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7659.100432] pending: neigh_periodic_work, check_lifetime [ 7659.103780] workqueue mm_percpu_wq: flags=0x8 [ 7659.105755] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7659.110647] pending: vmstat_update, lru_add_drain_per_cpu [ 7659.112122] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7659.119494] pending: vmstat_update [ 7659.121831] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7659.124435] pending: lru_add_drain_per_cpu BAR(48) [ 7659.127811] workqueue ipv6_addrconf: flags=0x40008 [ 7659.130677] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7659.134134] pending: addrconf_verify_work [ 7659.137254] workqueue knbd0-recv: flags=0x1a [ 7659.139056] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7659.143276] in-flight: 650:recv_work [ 7659.145415] workqueue ptlrpc_pinger: flags=0x2 [ 7659.146960] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7659.152967] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7659.158690] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5145s workers=4 idle: 36 466 29757 [ 7659.167103] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7649s workers=2 idle: 652 [ 7660.067098] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7660.073587] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7660.122020] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7660.135252] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7660.139543] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7660.149033] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7660.174809] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7660.181037] RAX: 0000000000002320 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7660.188528] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7660.191937] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7660.198315] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7660.205750] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7660.209741] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7660.213542] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7660.216299] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7660.220549] Call Trace: [ 7660.221448] [ 7660.222150] ? show_regs.cold.9+0x22/0x2f [ 7660.223739] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7660.230467] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7660.234381] ? watchdog+0x40/0x40 [ 7660.235553] ? hrtimer_interrupt+0x128/0x2c0 [ 7660.237428] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7660.238976] ? apic_timer_interrupt+0xf/0x20 [ 7660.243893] [ 7660.245100] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7660.250462] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7660.252663] do_raw_spin_lock+0x177/0x190 [ 7660.257102] _raw_spin_lock+0x21/0x30 [ 7660.259058] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7660.264164] ? do_raw_spin_unlock+0x75/0x190 [ 7660.266820] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7660.271020] ? do_wait_intr+0xf0/0xf0 [ 7660.272283] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7660.277544] mdc_reint+0x2d/0x110 [mdc] [ 7660.281804] mdc_unlink+0x658/0x870 [mdc] [ 7660.283514] lmv_unlink+0x34d/0xbd0 [lmv] [ 7660.298024] ll_unlink+0x24d/0x7f0 [lustre] [ 7660.300804] vfs_unlink+0x202/0x330 [ 7660.304732] do_unlinkat+0x2fa/0x460 [ 7660.306099] __x64_sys_unlink+0x37/0x50 [ 7660.312453] do_syscall_64+0xc1/0x440 [ 7660.314034] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7660.315365] RIP: 0033:0x7f83e5f5c10b [ 7660.318994] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7660.330704] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7660.338061] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7660.342327] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7660.346799] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7660.354068] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7660.357688] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7664.067648] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 7664.076071] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7664.095797] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7664.101163] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7664.105210] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7664.108519] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7664.111783] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7664.119903] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7664.122633] RAX: 0000000000007984 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7664.125698] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7664.128787] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7664.131729] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7664.135108] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7664.138465] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7664.142463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7664.145657] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7664.148904] Call Trace: [ 7664.150382] [ 7664.151409] ? show_regs.cold.9+0x22/0x2f [ 7664.153177] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7664.155674] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7664.157729] ? watchdog+0x40/0x40 [ 7664.159577] ? hrtimer_interrupt+0x128/0x2c0 [ 7664.161767] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7664.164338] ? apic_timer_interrupt+0xf/0x20 [ 7664.166203] [ 7664.167224] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7664.169668] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7664.172374] do_raw_spin_lock+0x177/0x190 [ 7664.174388] _raw_spin_lock+0x21/0x30 [ 7664.176208] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7664.179339] process_one_work+0x2c8/0x700 [ 7664.181206] worker_thread+0x296/0x6e0 [ 7664.183184] ? rescuer_thread+0x570/0x570 [ 7664.185262] kthread+0x1d1/0x200 [ 7664.186908] ? set_kthread_struct+0x70/0x70 [ 7664.189232] ret_from_fork+0x1f/0x30 [ 7676.059014] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 7676.067775] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7676.138175] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7676.145648] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7676.150099] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7676.153075] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7676.165432] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7676.172247] RAX: 0000000000002991 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7676.179968] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7676.191257] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7676.199614] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7676.204434] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7676.213720] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7676.220329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7676.227395] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7676.233199] Call Trace: [ 7676.235077] [ 7676.237989] ? show_regs.cold.9+0x22/0x2f [ 7676.242031] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7676.249189] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7676.253117] ? watchdog+0x40/0x40 [ 7676.256078] ? hrtimer_interrupt+0x128/0x2c0 [ 7676.260583] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7676.262461] ? apic_timer_interrupt+0xf/0x20 [ 7676.268635] [ 7676.269994] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7676.272896] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7676.276120] do_raw_spin_lock+0x177/0x190 [ 7676.278822] _raw_spin_lock+0x21/0x30 [ 7676.282153] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7676.287080] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7676.293091] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7676.299748] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7676.302593] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7676.305177] ? woken_wake_function+0x30/0x30 [ 7676.311807] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7676.317879] kthread+0x1d1/0x200 [ 7676.324474] ? set_kthread_struct+0x70/0x70 [ 7676.329920] ret_from_fork+0x1f/0x30 [ 7684.057018] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 7684.066827] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7684.108219] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7684.113778] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7684.117995] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7684.120707] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7684.131067] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7684.134460] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7684.138092] RDX: 0000000000005d73 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7684.141348] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7684.144664] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7684.148193] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7684.151544] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7684.155057] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7684.157883] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7684.160956] Call Trace: [ 7684.161783] [ 7684.162532] ? show_regs.cold.9+0x22/0x2f [ 7684.163992] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7684.165720] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7684.167328] ? watchdog+0x40/0x40 [ 7684.169028] ? hrtimer_interrupt+0x128/0x2c0 [ 7684.170863] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7684.173475] ? apic_timer_interrupt+0xf/0x20 [ 7684.175390] [ 7684.176496] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7684.179064] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7684.181791] do_raw_spin_lock+0x177/0x190 [ 7684.184673] _raw_spin_lock+0x21/0x30 [ 7684.186450] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7684.189056] obd_mod_free+0x46/0x140 [ptlrpc] [ 7684.191397] mdc_commit_open+0x91/0xb0 [mdc] [ 7684.193585] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7684.196503] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7684.199379] after_reply+0xe2c/0x1780 [ptlrpc] [ 7684.201718] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7684.204380] ? __next_timer_interrupt+0x160/0x160 [ 7684.206931] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7684.209339] ? do_wait_intr+0xf0/0xf0 [ 7684.210859] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7684.213604] kthread+0x1d1/0x200 [ 7684.215147] ? set_kthread_struct+0x70/0x70 [ 7684.217207] ret_from_fork+0x1f/0x30 [ 7688.062454] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 7688.066066] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7688.078822] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7688.081730] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7688.083789] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7688.085213] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7688.090435] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7688.093043] RAX: 000000000000272f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7688.095528] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7688.098104] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7688.100768] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7688.103085] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7688.105146] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7688.107759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7688.109784] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7688.112513] Call Trace: [ 7688.113425] [ 7688.114165] ? show_regs.cold.9+0x22/0x2f [ 7688.115607] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7688.117164] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7688.118644] ? watchdog+0x40/0x40 [ 7688.120218] ? hrtimer_interrupt+0x128/0x2c0 [ 7688.122463] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7688.125720] ? apic_timer_interrupt+0xf/0x20 [ 7688.128448] [ 7688.130343] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7688.134759] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7688.139086] do_raw_spin_lock+0x177/0x190 [ 7688.141252] _raw_spin_lock+0x21/0x30 [ 7688.142869] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7688.145195] ? do_raw_spin_unlock+0x75/0x190 [ 7688.147655] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7688.149961] ? do_wait_intr+0xf0/0xf0 [ 7688.151683] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7688.154005] mdc_reint+0x2d/0x110 [mdc] [ 7688.155749] mdc_unlink+0x658/0x870 [mdc] [ 7688.157421] lmv_unlink+0x34d/0xbd0 [lmv] [ 7688.159502] ll_unlink+0x24d/0x7f0 [lustre] [ 7688.161419] vfs_unlink+0x202/0x330 [ 7688.162943] do_unlinkat+0x2fa/0x460 [ 7688.164428] __x64_sys_unlink+0x37/0x50 [ 7688.166213] do_syscall_64+0xc1/0x440 [ 7688.167831] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7688.169860] RIP: 0033:0x7f83e5f5c10b [ 7688.171563] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7688.180243] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7688.182943] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7688.186351] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7688.189079] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7688.192007] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7688.195841] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7689.184121] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5175s! [ 7689.186196] Showing busy workqueues and worker pools: [ 7689.187498] workqueue events: flags=0x0 [ 7689.188518] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7689.190355] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7689.192625] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7689.194147] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7689.196137] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7689.197933] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7689.200047] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7689.201797] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7689.204191] workqueue events_long: flags=0x0 [ 7689.205253] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7689.206997] pending: lu_objects_flush [obdclass] [ 7689.208507] workqueue events_unbound: flags=0x2 [ 7689.209765] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7689.211396] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7689.212944] workqueue events_power_efficient: flags=0x80 [ 7689.214175] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7689.216117] pending: neigh_periodic_work, do_cache_clean [ 7689.217685] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7689.219405] pending: neigh_periodic_work, check_lifetime [ 7689.220867] workqueue mm_percpu_wq: flags=0x8 [ 7689.222207] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7689.224291] pending: vmstat_update, lru_add_drain_per_cpu [ 7689.225958] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7689.227872] pending: vmstat_update [ 7689.228899] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7689.230742] pending: lru_add_drain_per_cpu BAR(48) [ 7689.232179] workqueue ipv6_addrconf: flags=0x40008 [ 7689.233453] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7689.235205] pending: addrconf_verify_work [ 7689.236402] workqueue knbd0-recv: flags=0x1a [ 7689.237440] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7689.239132] in-flight: 650:recv_work [ 7689.240168] workqueue ptlrpc_pinger: flags=0x2 [ 7689.241275] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7689.242864] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7689.244678] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5176s workers=4 idle: 36 466 29757 [ 7689.246708] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7679s workers=2 idle: 652 [ 7692.060011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 7692.063028] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7692.083413] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7692.087831] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7692.091054] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7692.093805] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7692.097465] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7692.105231] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7692.108635] RAX: 0000000000003a50 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7692.111650] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7692.114429] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7692.118195] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7692.121833] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7692.124734] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7692.128105] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7692.130321] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7692.133203] Call Trace: [ 7692.134314] [ 7692.135124] ? show_regs.cold.9+0x22/0x2f [ 7692.136735] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7692.138488] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7692.140499] ? watchdog+0x40/0x40 [ 7692.141998] ? hrtimer_interrupt+0x128/0x2c0 [ 7692.143762] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7692.145925] ? apic_timer_interrupt+0xf/0x20 [ 7692.148296] [ 7692.149179] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7692.150830] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7692.152283] do_raw_spin_lock+0x177/0x190 [ 7692.154365] _raw_spin_lock+0x21/0x30 [ 7692.155852] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7692.158906] process_one_work+0x2c8/0x700 [ 7692.160610] worker_thread+0x296/0x6e0 [ 7692.162079] ? rescuer_thread+0x570/0x570 [ 7692.163728] kthread+0x1d1/0x200 [ 7692.165113] ? set_kthread_struct+0x70/0x70 [ 7692.166858] ret_from_fork+0x1f/0x30 [ 7704.059070] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 7704.062705] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7704.110234] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7704.116207] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7704.120793] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7704.123281] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7704.134074] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7704.137923] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7704.144166] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7704.150806] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7704.155287] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7704.162128] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7704.164140] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7704.171859] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7704.175698] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7704.180599] Call Trace: [ 7704.181848] [ 7704.192534] ? show_regs.cold.9+0x22/0x2f [ 7704.194061] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7704.200578] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7704.212161] ? watchdog+0x40/0x40 [ 7704.216696] ? hrtimer_interrupt+0x128/0x2c0 [ 7704.220671] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7704.225943] ? apic_timer_interrupt+0xf/0x20 [ 7704.228721] [ 7704.231454] ? native_safe_halt+0x2e/0x50 [ 7704.233790] kvm_wait+0xcd/0xe0 [ 7704.237528] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7704.240690] do_raw_spin_lock+0x177/0x190 [ 7704.242987] _raw_spin_lock+0x21/0x30 [ 7704.245738] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7704.248791] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7704.255406] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7704.260385] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7704.265211] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7704.268408] ? woken_wake_function+0x30/0x30 [ 7704.271880] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7704.276233] kthread+0x1d1/0x200 [ 7704.279531] ? set_kthread_struct+0x70/0x70 [ 7704.283964] ret_from_fork+0x1f/0x30 [ 7716.058584] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 7716.062598] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 7716.065743] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7716.065827] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7716.065835] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7716.065838] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7716.065850] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7716.065855] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7716.065954] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7716.065957] RDX: 0000000000007729 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7716.065959] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7716.065961] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7716.065963] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7716.065967] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7716.065970] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7716.065972] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7716.065980] Call Trace: [ 7716.065987] [ 7716.065992] ? show_regs.cold.9+0x22/0x2f [ 7716.066004] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7716.066010] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7716.066014] ? watchdog+0x40/0x40 [ 7716.066019] ? hrtimer_interrupt+0x128/0x2c0 [ 7716.066023] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7716.066031] ? apic_timer_interrupt+0xf/0x20 [ 7716.066036] [ 7716.066039] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7716.066044] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7716.066049] do_raw_spin_lock+0x177/0x190 [ 7716.066054] _raw_spin_lock+0x21/0x30 [ 7716.066060] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7716.074421] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7716.074779] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7716.074787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7716.074790] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7716.074804] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7716.074809] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7716.074813] RAX: 0000000000007a91 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7716.074816] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7716.074818] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7716.074820] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7716.074823] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7716.074826] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7716.074829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7716.074831] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7716.074840] Call Trace: [ 7716.074845] [ 7716.074851] ? show_regs.cold.9+0x22/0x2f [ 7716.074861] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7716.074867] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7716.074871] ? watchdog+0x40/0x40 [ 7716.074875] ? hrtimer_interrupt+0x128/0x2c0 [ 7716.074878] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7716.074885] ? apic_timer_interrupt+0xf/0x20 [ 7716.074889] [ 7716.074891] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7716.074895] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7716.075070] do_raw_spin_lock+0x177/0x190 [ 7716.075075] _raw_spin_lock+0x21/0x30 [ 7716.075081] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7716.075382] ? do_raw_spin_unlock+0x75/0x190 [ 7716.075468] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7716.122521] obd_mod_free+0x46/0x140 [ptlrpc] [ 7716.128528] ? do_wait_intr+0xf0/0xf0 [ 7716.128550] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7716.129417] mdc_reint+0x2d/0x110 [mdc] [ 7716.129483] mdc_unlink+0x658/0x870 [mdc] [ 7716.129508] lmv_unlink+0x34d/0xbd0 [lmv] [ 7716.129538] ll_unlink+0x24d/0x7f0 [lustre] [ 7716.129775] vfs_unlink+0x202/0x330 [ 7716.129813] do_unlinkat+0x2fa/0x460 [ 7716.129818] __x64_sys_unlink+0x37/0x50 [ 7716.129822] do_syscall_64+0xc1/0x440 [ 7716.129836] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7716.129847] RIP: 0033:0x7f83e5f5c10b [ 7716.129894] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7716.129899] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7716.129904] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7716.129907] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7716.129909] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7716.129912] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7716.129914] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7716.437012] mdc_commit_open+0x91/0xb0 [mdc] [ 7716.441146] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7716.444360] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7716.447467] after_reply+0xe2c/0x1780 [ptlrpc] [ 7716.449967] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7716.452660] ? __next_timer_interrupt+0x160/0x160 [ 7716.454055] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7716.455644] ? do_wait_intr+0xf0/0xf0 [ 7716.456752] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7716.459303] kthread+0x1d1/0x200 [ 7716.461804] ? set_kthread_struct+0x70/0x70 [ 7716.463709] ret_from_fork+0x1f/0x30 [ 7719.393046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5206s! [ 7719.397311] Showing busy workqueues and worker pools: [ 7719.399795] workqueue events: flags=0x0 [ 7719.402160] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7719.411794] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7719.420133] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7719.431622] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7719.438222] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7719.446116] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7719.462163] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7719.472671] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7719.483053] workqueue events_long: flags=0x0 [ 7719.486027] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7719.493376] pending: lu_objects_flush [obdclass] [ 7719.497968] workqueue events_unbound: flags=0x2 [ 7719.509956] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7719.515648] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7719.520588] workqueue events_power_efficient: flags=0x80 [ 7719.526913] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7719.533379] pending: neigh_periodic_work, do_cache_clean [ 7719.549253] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7719.557141] pending: neigh_periodic_work, check_lifetime [ 7719.564097] workqueue mm_percpu_wq: flags=0x8 [ 7719.568173] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7719.575506] pending: vmstat_update, lru_add_drain_per_cpu [ 7719.580207] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7719.584503] pending: vmstat_update [ 7719.588595] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7719.597338] pending: lru_add_drain_per_cpu BAR(48) [ 7719.600710] workqueue ipv6_addrconf: flags=0x40008 [ 7719.611181] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7719.617027] pending: addrconf_verify_work [ 7719.622186] workqueue knbd0-recv: flags=0x1a [ 7719.628021] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7719.632727] in-flight: 650:recv_work [ 7719.635082] workqueue ptlrpc_pinger: flags=0x2 [ 7719.639134] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7719.646027] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7719.650686] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5206s workers=4 idle: 36 466 29757 [ 7719.658686] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7709s workers=2 idle: 652 [ 7720.060025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 7720.063584] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7720.092643] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7720.103010] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7720.114835] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7720.118611] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7720.126840] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7720.136747] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7720.139647] RAX: 0000000000002661 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7720.142467] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7720.145193] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7720.148058] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7720.150901] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7720.153622] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7720.156664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7720.158321] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7720.160616] Call Trace: [ 7720.161716] [ 7720.162442] ? show_regs.cold.9+0x22/0x2f [ 7720.163598] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7720.165055] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7720.166529] ? watchdog+0x40/0x40 [ 7720.167642] ? hrtimer_interrupt+0x128/0x2c0 [ 7720.171084] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7720.173496] ? apic_timer_interrupt+0xf/0x20 [ 7720.176389] [ 7720.177245] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7720.180664] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7720.185848] do_raw_spin_lock+0x177/0x190 [ 7720.188736] _raw_spin_lock+0x21/0x30 [ 7720.189787] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7720.192859] process_one_work+0x2c8/0x700 [ 7720.194890] worker_thread+0x296/0x6e0 [ 7720.201792] ? rescuer_thread+0x570/0x570 [ 7720.205587] kthread+0x1d1/0x200 [ 7720.208565] ? set_kthread_struct+0x70/0x70 [ 7720.211872] ret_from_fork+0x1f/0x30 [ 7732.061789] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 7732.073808] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7732.104323] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7732.109003] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7732.112568] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7732.114502] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7732.121758] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7732.127815] RAX: 0000000000002e17 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7732.140649] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7732.151638] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7732.155588] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7732.165882] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7732.181461] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7732.188992] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7732.198543] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7732.207519] Call Trace: [ 7732.208495] [ 7732.209162] ? show_regs.cold.9+0x22/0x2f [ 7732.214630] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7732.217493] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7732.220122] ? watchdog+0x40/0x40 [ 7732.221671] ? hrtimer_interrupt+0x128/0x2c0 [ 7732.229081] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7732.233830] ? apic_timer_interrupt+0xf/0x20 [ 7732.236593] [ 7732.242367] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7732.247124] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7732.255703] do_raw_spin_lock+0x177/0x190 [ 7732.259386] _raw_spin_lock+0x21/0x30 [ 7732.260779] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7732.266031] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7732.273119] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7732.278508] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7732.284928] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7732.289172] ? woken_wake_function+0x30/0x30 [ 7732.293090] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7732.297210] kthread+0x1d1/0x200 [ 7732.298762] ? set_kthread_struct+0x70/0x70 [ 7732.304493] ret_from_fork+0x1f/0x30 [ 7744.056072] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 7744.060117] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7744.067085] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 7744.080462] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7744.080477] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7744.080480] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7744.080495] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7744.080499] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7744.080505] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7744.080508] RDX: 0000000000006caa RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7744.080510] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7744.080512] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7744.080515] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7744.080518] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7744.080521] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7744.080523] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7744.080531] Call Trace: [ 7744.080538] [ 7744.080544] ? show_regs.cold.9+0x22/0x2f [ 7744.080555] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7744.080561] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7744.080566] ? watchdog+0x40/0x40 [ 7744.080571] ? hrtimer_interrupt+0x128/0x2c0 [ 7744.080575] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7744.080584] ? apic_timer_interrupt+0xf/0x20 [ 7744.080589] [ 7744.080592] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7744.080597] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7744.080603] do_raw_spin_lock+0x177/0x190 [ 7744.080607] _raw_spin_lock+0x21/0x30 [ 7744.080615] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7744.081041] obd_mod_free+0x46/0x140 [ptlrpc] [ 7744.081236] mdc_commit_open+0x91/0xb0 [mdc] [ 7744.081279] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7744.081484] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7744.084592] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7744.084922] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7744.084928] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7744.084932] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7744.084944] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7744.084948] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7744.084952] RAX: 0000000000007c32 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7744.084955] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7744.084957] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7744.084959] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7744.084961] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7744.084964] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7744.084967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7744.084969] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7744.084977] Call Trace: [ 7744.084983] [ 7744.084988] ? show_regs.cold.9+0x22/0x2f [ 7744.084997] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7744.085003] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7744.085007] ? watchdog+0x40/0x40 [ 7744.085012] ? hrtimer_interrupt+0x128/0x2c0 [ 7744.085016] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7744.085023] ? apic_timer_interrupt+0xf/0x20 [ 7744.085040] [ 7744.085043] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7744.085048] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7744.085053] do_raw_spin_lock+0x177/0x190 [ 7744.085058] _raw_spin_lock+0x21/0x30 [ 7744.085064] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7744.085321] ? do_raw_spin_unlock+0x75/0x190 [ 7744.085326] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7744.085525] ? do_wait_intr+0xf0/0xf0 [ 7744.085532] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7744.092356] after_reply+0xe2c/0x1780 [ptlrpc] [ 7744.096535] mdc_reint+0x2d/0x110 [mdc] [ 7744.096583] mdc_unlink+0x658/0x870 [mdc] [ 7744.096618] lmv_unlink+0x34d/0xbd0 [lmv] [ 7744.096890] ll_unlink+0x24d/0x7f0 [lustre] [ 7744.097003] vfs_unlink+0x202/0x330 [ 7744.097011] do_unlinkat+0x2fa/0x460 [ 7744.097017] __x64_sys_unlink+0x37/0x50 [ 7744.097020] do_syscall_64+0xc1/0x440 [ 7744.097027] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7744.097047] RIP: 0033:0x7f83e5f5c10b [ 7744.097056] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7744.097060] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7744.097065] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7744.097068] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7744.097070] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7744.097072] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7744.097074] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7744.385644] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7744.387940] ? __next_timer_interrupt+0x160/0x160 [ 7744.394332] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7744.396657] ? do_wait_intr+0xf0/0xf0 [ 7744.402282] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7744.408566] kthread+0x1d1/0x200 [ 7744.411489] ? set_kthread_struct+0x70/0x70 [ 7744.417627] ret_from_fork+0x1f/0x30 [ 7748.060026] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 7748.063568] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7748.088945] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7748.093748] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7748.097942] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7748.100726] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7748.103476] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7748.111495] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7748.114589] RAX: 0000000000001485 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7748.117627] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7748.121350] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7748.124269] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7748.127497] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7748.130780] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7748.134737] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7748.136964] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7748.143582] Call Trace: [ 7748.144770] [ 7748.145628] ? show_regs.cold.9+0x22/0x2f [ 7748.147588] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7748.149652] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7748.151740] ? watchdog+0x40/0x40 [ 7748.153904] ? hrtimer_interrupt+0x128/0x2c0 [ 7748.155709] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7748.157430] ? apic_timer_interrupt+0xf/0x20 [ 7748.159660] [ 7748.160573] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7748.162760] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7748.165211] do_raw_spin_lock+0x177/0x190 [ 7748.167155] _raw_spin_lock+0x21/0x30 [ 7748.169177] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7748.171848] process_one_work+0x2c8/0x700 [ 7748.173512] worker_thread+0x296/0x6e0 [ 7748.174979] ? rescuer_thread+0x570/0x570 [ 7748.176597] kthread+0x1d1/0x200 [ 7748.177929] ? set_kthread_struct+0x70/0x70 [ 7748.179489] ret_from_fork+0x1f/0x30 [ 7750.116131] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5236s! [ 7750.120370] Showing busy workqueues and worker pools: [ 7750.122933] workqueue events: flags=0x0 [ 7750.125107] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7750.128418] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7750.133382] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7750.137133] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7750.140349] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7750.143673] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7750.147390] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7750.150519] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7750.154857] workqueue events_long: flags=0x0 [ 7750.156628] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7750.159409] pending: lu_objects_flush [obdclass] [ 7750.161727] workqueue events_unbound: flags=0x2 [ 7750.164215] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7750.167504] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7750.170559] workqueue events_power_efficient: flags=0x80 [ 7750.172691] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7750.175870] pending: neigh_periodic_work, do_cache_clean [ 7750.178450] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7750.181583] pending: neigh_periodic_work, check_lifetime [ 7750.184172] workqueue mm_percpu_wq: flags=0x8 [ 7750.185965] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7750.188758] pending: vmstat_update, lru_add_drain_per_cpu [ 7750.191736] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7750.195265] pending: vmstat_update [ 7750.197682] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7750.200569] pending: lru_add_drain_per_cpu BAR(48) [ 7750.202156] workqueue ipv6_addrconf: flags=0x40008 [ 7750.203552] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7750.205908] pending: addrconf_verify_work [ 7750.207254] workqueue knbd0-recv: flags=0x1a [ 7750.208658] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7750.210752] in-flight: 650:recv_work [ 7750.211930] workqueue ptlrpc_pinger: flags=0x2 [ 7750.213447] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7750.215141] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7750.217441] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5237s workers=4 idle: 36 466 29757 [ 7750.219728] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7740s workers=2 idle: 652 [ 7760.059030] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 7760.065567] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7760.105049] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7760.115263] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7760.123660] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7760.129708] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7760.141219] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7760.147972] RAX: 0000000000007389 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7760.156159] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7760.159879] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7760.166473] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7760.170247] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7760.173454] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7760.179425] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7760.182276] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7760.186043] Call Trace: [ 7760.189636] [ 7760.191223] ? show_regs.cold.9+0x22/0x2f [ 7760.195295] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7760.197493] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7760.203063] ? watchdog+0x40/0x40 [ 7760.204559] ? hrtimer_interrupt+0x128/0x2c0 [ 7760.210429] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7760.218071] ? apic_timer_interrupt+0xf/0x20 [ 7760.220551] [ 7760.222294] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7760.229433] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7760.233300] do_raw_spin_lock+0x177/0x190 [ 7760.237743] _raw_spin_lock+0x21/0x30 [ 7760.239561] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7760.243312] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7760.248856] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7760.252482] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7760.257823] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7760.261601] ? woken_wake_function+0x30/0x30 [ 7760.264759] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7760.270786] kthread+0x1d1/0x200 [ 7760.272362] ? set_kthread_struct+0x70/0x70 [ 7760.276395] ret_from_fork+0x1f/0x30 [ 7772.062813] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 7772.066232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7772.089690] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7772.095898] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7772.099326] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7772.101948] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7772.109843] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7772.112997] RAX: 0000000000005a44 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7772.115752] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7772.120247] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7772.123452] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7772.127039] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7772.130255] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7772.133048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7772.134658] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7772.136830] Call Trace: [ 7772.137900] [ 7772.138451] ? show_regs.cold.9+0x22/0x2f [ 7772.139878] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7772.141522] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7772.143203] ? watchdog+0x40/0x40 [ 7772.145863] ? hrtimer_interrupt+0x128/0x2c0 [ 7772.147856] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7772.150048] ? apic_timer_interrupt+0xf/0x20 [ 7772.152177] [ 7772.153226] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7772.155875] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7772.158327] do_raw_spin_lock+0x177/0x190 [ 7772.159875] _raw_spin_lock+0x21/0x30 [ 7772.161641] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7772.164199] ? do_raw_spin_unlock+0x75/0x190 [ 7772.165896] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7772.168188] ? do_wait_intr+0xf0/0xf0 [ 7772.170246] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7772.172997] mdc_reint+0x2d/0x110 [mdc] [ 7772.174955] mdc_unlink+0x658/0x870 [mdc] [ 7772.177450] lmv_unlink+0x34d/0xbd0 [lmv] [ 7772.179141] ll_unlink+0x24d/0x7f0 [lustre] [ 7772.181158] vfs_unlink+0x202/0x330 [ 7772.182659] do_unlinkat+0x2fa/0x460 [ 7772.184438] __x64_sys_unlink+0x37/0x50 [ 7772.186273] do_syscall_64+0xc1/0x440 [ 7772.188068] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7772.190417] RIP: 0033:0x7f83e5f5c10b [ 7772.191918] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7772.201239] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7772.204828] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7772.207135] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7772.210270] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7772.213891] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7772.216926] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7776.056055] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 7776.060016] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 7776.066267] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata [ 7776.069034] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) [ 7776.109500] dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7776.109529] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7776.109537] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7776.109540] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7776.109555] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7776.109559] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7776.109564] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7776.109566] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7776.109568] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7776.109570] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7776.109572] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7776.109575] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7776.109578] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7776.109580] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7776.109589] Call Trace: [ 7776.109596] [ 7776.109602] ? show_regs.cold.9+0x22/0x2f [ 7776.109613] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7776.109620] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7776.109624] ? watchdog+0x40/0x40 [ 7776.109640] ? hrtimer_interrupt+0x128/0x2c0 [ 7776.109644] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7776.109651] ? apic_timer_interrupt+0xf/0x20 [ 7776.109656] [ 7776.109659] ? native_safe_halt+0x2e/0x50 [ 7776.109664] kvm_wait+0xcd/0xe0 [ 7776.109671] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7776.109677] do_raw_spin_lock+0x177/0x190 [ 7776.109681] _raw_spin_lock+0x21/0x30 [ 7776.109688] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7776.116068] fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7776.116164] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7776.116170] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7776.116177] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7776.116540] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7776.116550] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7776.116553] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7776.116558] RAX: 0000000000004b4c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7776.116560] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7776.116562] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7776.116564] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7776.116566] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7776.116570] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7776.116572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7776.116574] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7776.116581] Call Trace: [ 7776.116586] [ 7776.116590] ? show_regs.cold.9+0x22/0x2f [ 7776.116599] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7776.116605] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7776.116609] ? watchdog+0x40/0x40 [ 7776.116614] ? hrtimer_interrupt+0x128/0x2c0 [ 7776.116617] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7776.116999] ? apic_timer_interrupt+0xf/0x20 [ 7776.117004] [ 7776.117007] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7776.117012] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7776.117020] do_raw_spin_lock+0x177/0x190 [ 7776.117025] _raw_spin_lock+0x21/0x30 [ 7776.117030] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7776.117270] process_one_work+0x2c8/0x700 [ 7776.117277] worker_thread+0x296/0x6e0 [ 7776.117281] ? rescuer_thread+0x570/0x570 [ 7776.117285] kthread+0x1d1/0x200 [ 7776.117292] ? set_kthread_struct+0x70/0x70 [ 7776.117296] ret_from_fork+0x1f/0x30 [ 7776.471417] obd_mod_free+0x46/0x140 [ptlrpc] [ 7776.477353] mdc_commit_open+0x91/0xb0 [mdc] [ 7776.480359] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7776.485156] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7776.489099] after_reply+0xe2c/0x1780 [ptlrpc] [ 7776.492201] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7776.495443] ? __next_timer_interrupt+0x160/0x160 [ 7776.498264] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7776.500894] ? do_wait_intr+0xf0/0xf0 [ 7776.502273] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7776.506400] kthread+0x1d1/0x200 [ 7776.509992] ? set_kthread_struct+0x70/0x70 [ 7776.511505] ret_from_fork+0x1f/0x30 [ 7780.320984] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5267s! [ 7780.331822] Showing busy workqueues and worker pools: [ 7780.335027] workqueue events: flags=0x0 [ 7780.342174] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7780.346649] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7780.354987] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7780.362511] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7780.365209] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7780.368346] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7780.372157] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7780.374609] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7780.380378] workqueue events_long: flags=0x0 [ 7780.384018] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7780.390314] pending: lu_objects_flush [obdclass] [ 7780.395195] workqueue events_unbound: flags=0x2 [ 7780.397973] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7780.404298] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7780.412927] workqueue events_power_efficient: flags=0x80 [ 7780.418337] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7780.428642] pending: neigh_periodic_work, do_cache_clean [ 7780.432937] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7780.442035] pending: neigh_periodic_work, check_lifetime [ 7780.444850] workqueue mm_percpu_wq: flags=0x8 [ 7780.448430] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7780.456423] pending: vmstat_update, lru_add_drain_per_cpu [ 7780.460069] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7780.463275] pending: vmstat_update [ 7780.465161] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7780.468957] pending: lru_add_drain_per_cpu BAR(48) [ 7780.471099] workqueue ipv6_addrconf: flags=0x40008 [ 7780.476336] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7780.480834] pending: addrconf_verify_work [ 7780.482667] workqueue knbd0-recv: flags=0x1a [ 7780.486988] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7780.491622] in-flight: 650:recv_work [ 7780.500053] workqueue ptlrpc_pinger: flags=0x2 [ 7780.502088] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7780.506490] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7780.514726] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5267s workers=4 idle: 36 466 29757 [ 7780.522639] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7770s workers=2 idle: 652 [ 7788.059028] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 7788.062038] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7788.079825] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7788.084586] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7788.087903] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7788.089870] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7788.097250] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7788.100293] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7788.103484] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7788.106231] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7788.109729] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7788.112989] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7788.116098] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7788.119220] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7788.121180] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7788.124443] Call Trace: [ 7788.125352] [ 7788.126183] ? show_regs.cold.9+0x22/0x2f [ 7788.128104] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7788.132184] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7788.137163] ? watchdog+0x40/0x40 [ 7788.138684] ? hrtimer_interrupt+0x128/0x2c0 [ 7788.143476] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7788.144791] ? apic_timer_interrupt+0xf/0x20 [ 7788.147305] [ 7788.148288] ? native_safe_halt+0x2e/0x50 [ 7788.150040] kvm_wait+0xcd/0xe0 [ 7788.151657] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7788.153758] do_raw_spin_lock+0x177/0x190 [ 7788.155490] _raw_spin_lock+0x21/0x30 [ 7788.156983] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7788.161272] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7788.165006] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7788.171345] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7788.175585] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7788.178175] ? woken_wake_function+0x30/0x30 [ 7788.180926] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7788.184258] kthread+0x1d1/0x200 [ 7788.187083] ? set_kthread_struct+0x70/0x70 [ 7788.188126] ret_from_fork+0x1f/0x30 [ 7800.062726] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7800.070092] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7800.121721] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7800.140715] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7800.147392] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7800.159819] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7800.184895] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7800.198179] RAX: 0000000000001b8a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7800.207671] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7800.214888] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7800.222361] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7800.224803] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7800.239342] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7800.255476] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7800.271188] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7800.287575] Call Trace: [ 7800.288466] [ 7800.289126] ? show_regs.cold.9+0x22/0x2f [ 7800.299628] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7800.301146] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7800.308729] ? watchdog+0x40/0x40 [ 7800.311257] ? hrtimer_interrupt+0x128/0x2c0 [ 7800.317794] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7800.329443] ? apic_timer_interrupt+0xf/0x20 [ 7800.330724] [ 7800.342706] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7800.344606] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7800.354977] do_raw_spin_lock+0x177/0x190 [ 7800.356198] _raw_spin_lock+0x21/0x30 [ 7800.357276] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7800.366022] ? do_raw_spin_unlock+0x75/0x190 [ 7800.373368] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7800.375476] ? do_wait_intr+0xf0/0xf0 [ 7800.379231] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7800.387173] mdc_reint+0x2d/0x110 [mdc] [ 7800.397694] mdc_unlink+0x658/0x870 [mdc] [ 7800.399015] lmv_unlink+0x34d/0xbd0 [lmv] [ 7800.406102] ll_unlink+0x24d/0x7f0 [lustre] [ 7800.417781] vfs_unlink+0x202/0x330 [ 7800.419374] do_unlinkat+0x2fa/0x460 [ 7800.426861] __x64_sys_unlink+0x37/0x50 [ 7800.436824] do_syscall_64+0xc1/0x440 [ 7800.438191] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7800.444914] RIP: 0033:0x7f83e5f5c10b [ 7800.455212] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7800.487060] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7800.495443] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7800.505352] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7800.515826] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7800.519732] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7800.528793] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7801.096858] rcu: INFO: rcu_sched self-detected stall on CPU [ 7801.099660] rcu: 0-....: (3711570 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 7801.109572] (t=5820150 jiffies g=262441 q=58862) [ 7801.115649] rcu: rcu_sched kthread starved for 5289343 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 7801.130019] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7801.138247] rcu: RCU grace-period kthread stack dump: [ 7801.145027] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7801.154055] Call Trace: [ 7801.160034] __schedule+0x351/0xcb0 [ 7801.163392] schedule+0xc0/0x180 [ 7801.164613] schedule_timeout+0xb4/0x190 [ 7801.170067] ? do_raw_spin_unlock+0x75/0x190 [ 7801.176657] ? __next_timer_interrupt+0x160/0x160 [ 7801.179217] ? prepare_to_swait_event+0xf3/0x140 [ 7801.184059] rcu_gp_kthread+0x8dd/0x15b0 [ 7801.188525] ? force_qs_rnp+0x2c0/0x2c0 [ 7801.191447] kthread+0x1d1/0x200 [ 7801.192689] ? set_kthread_struct+0x70/0x70 [ 7801.196201] ret_from_fork+0x1f/0x30 [ 7801.201037] rcu: Stack dump where RCU GP kthread last ran: [ 7801.205953] Sending NMI from CPU 0 to CPUs 1: [ 7801.213052] NMI backtrace for cpu 1 [ 7801.213060] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7801.213062] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7801.213064] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7801.213065] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7801.213067] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 [ 7801.213070] RAX: 0000000000000e91 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7801.213072] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7801.213073] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7801.213075] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7801.213076] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7801.213078] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7801.213080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7801.213081] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7801.213082] Call Trace: [ 7801.213083] [ 7801.213084] ? show_regs.cold.9+0x22/0x2f [ 7801.213085] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7801.213087] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7801.213088] ? nmi_handle+0xbd/0x210 [ 7801.213089] ? default_do_nmi+0x4b/0x1b0 [ 7801.213091] ? do_nmi+0x280/0x380 [ 7801.213092] ? end_repeat_nmi+0x16/0x1f [ 7801.213093] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7801.213094] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7801.213096] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7801.213097] [ 7801.213098] do_raw_spin_lock+0x177/0x190 [ 7801.213099] _raw_spin_lock+0x21/0x30 [ 7801.213101] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7801.213102] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7801.213103] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7801.213105] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7801.213106] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7801.213107] ? woken_wake_function+0x30/0x30 [ 7801.213109] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7801.213110] kthread+0x1d1/0x200 [ 7801.213111] ? set_kthread_struct+0x70/0x70 [ 7801.213112] ret_from_fork+0x1f/0x30 [ 7801.216530] NMI backtrace for cpu 0 [ 7801.468152] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7801.477105] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7801.482321] Call Trace: [ 7801.483671] [ 7801.490005] ? dump_stack+0xbb/0x10e [ 7801.494401] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7801.500905] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7801.508367] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7801.515723] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7801.517688] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7801.523690] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7801.528374] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7801.531690] ? update_process_times+0x6d/0xc0 [ 7801.534407] ? tick_sched_handle.isra.18+0x27/0x90 [ 7801.536583] ? tick_sched_timer+0x53/0xb0 [ 7801.538238] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7801.540153] ? tick_sched_handle.isra.18+0x90/0x90 [ 7801.542286] ? hrtimer_interrupt+0x128/0x2c0 [ 7801.544189] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7801.547288] ? apic_timer_interrupt+0xf/0x20 [ 7801.548839] [ 7801.549599] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7801.552364] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7801.557209] ? do_raw_spin_lock+0x177/0x190 [ 7801.559079] ? _raw_spin_lock+0x21/0x30 [ 7801.561087] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7801.565979] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7801.568922] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7801.570941] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7801.574376] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7801.579292] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7801.581644] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7801.584433] ? __next_timer_interrupt+0x160/0x160 [ 7801.588123] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7801.590866] ? do_wait_intr+0xf0/0xf0 [ 7801.593069] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7801.595792] ? kthread+0x1d1/0x200 [ 7801.603429] ? set_kthread_struct+0x70/0x70 [ 7801.607737] ? ret_from_fork+0x1f/0x30 [ 7804.060371] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 7804.065288] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7804.105889] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7804.111057] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7804.114759] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7804.118165] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7804.120824] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7804.131684] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7804.143919] RAX: 0000000000007f85 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7804.150503] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7804.154396] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7804.158497] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7804.162120] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7804.168880] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7804.171725] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7804.174884] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7804.178651] Call Trace: [ 7804.179225] [ 7804.180297] ? show_regs.cold.9+0x22/0x2f [ 7804.181581] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7804.185726] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7804.189241] ? watchdog+0x40/0x40 [ 7804.190721] ? hrtimer_interrupt+0x128/0x2c0 [ 7804.196043] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7804.199337] ? apic_timer_interrupt+0xf/0x20 [ 7804.203297] [ 7804.204680] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7804.211364] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7804.215058] do_raw_spin_lock+0x177/0x190 [ 7804.216777] _raw_spin_lock+0x21/0x30 [ 7804.220948] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7804.223694] process_one_work+0x2c8/0x700 [ 7804.225830] worker_thread+0x296/0x6e0 [ 7804.230279] ? rescuer_thread+0x570/0x570 [ 7804.231899] kthread+0x1d1/0x200 [ 7804.233471] ? set_kthread_struct+0x70/0x70 [ 7804.235657] ret_from_fork+0x1f/0x30 [ 7811.048149] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5297s! [ 7811.052875] Showing busy workqueues and worker pools: [ 7811.054791] workqueue events: flags=0x0 [ 7811.058644] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7811.061341] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7811.064653] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7811.066547] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7811.068744] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7811.070845] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7811.072924] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7811.074820] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7811.077331] workqueue events_long: flags=0x0 [ 7811.078451] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7811.080363] pending: lu_objects_flush [obdclass] [ 7811.081995] workqueue events_unbound: flags=0x2 [ 7811.083310] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7811.085002] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7811.086521] workqueue events_power_efficient: flags=0x80 [ 7811.087815] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7811.089509] pending: neigh_periodic_work, do_cache_clean [ 7811.091115] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7811.094026] pending: neigh_periodic_work, check_lifetime [ 7811.095340] workqueue mm_percpu_wq: flags=0x8 [ 7811.096827] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7811.098469] pending: vmstat_update, lru_add_drain_per_cpu [ 7811.100095] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7811.101678] pending: vmstat_update [ 7811.102573] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7811.104734] pending: lru_add_drain_per_cpu BAR(48) [ 7811.106248] workqueue ipv6_addrconf: flags=0x40008 [ 7811.107563] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7811.109325] pending: addrconf_verify_work [ 7811.110636] workqueue knbd0-recv: flags=0x1a [ 7811.111982] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7811.113778] in-flight: 650:recv_work [ 7811.114773] workqueue ptlrpc_pinger: flags=0x2 [ 7811.116204] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7811.118659] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7811.120584] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5297s workers=4 idle: 36 466 29757 [ 7811.122728] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7801s workers=2 idle: 652 [ 7828.059787] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 7828.063897] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7828.067142] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7828.067222] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7828.067230] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7828.067233] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7828.067247] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7828.067251] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7828.067255] RAX: 000000000000461f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7828.067258] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7828.067260] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7828.067263] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7828.067266] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7828.067270] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7828.067273] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7828.067275] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7828.067283] Call Trace: [ 7828.067288] [ 7828.067294] ? show_regs.cold.9+0x22/0x2f [ 7828.067302] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7828.067308] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7828.067312] ? watchdog+0x40/0x40 [ 7828.067316] ? hrtimer_interrupt+0x128/0x2c0 [ 7828.067321] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7828.067329] ? apic_timer_interrupt+0xf/0x20 [ 7828.067334] [ 7828.067337] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7828.067342] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7828.067348] do_raw_spin_lock+0x177/0x190 [ 7828.067352] _raw_spin_lock+0x21/0x30 [ 7828.067357] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7828.084099] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7828.084200] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7828.084207] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7828.084211] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 7828.084236] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 7828.084243] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7828.084249] RAX: 00000000000014c6 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7828.084252] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7828.084254] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7828.084256] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7828.084259] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7828.084262] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7828.084266] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7828.084268] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7828.084276] Call Trace: [ 7828.084285] [ 7828.084291] ? show_regs.cold.9+0x22/0x2f [ 7828.084362] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7828.084369] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7828.084373] ? watchdog+0x40/0x40 [ 7828.084378] ? hrtimer_interrupt+0x128/0x2c0 [ 7828.084382] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7828.084392] ? apic_timer_interrupt+0xf/0x20 [ 7828.084400] [ 7828.084403] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 7828.084408] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7828.084413] do_raw_spin_lock+0x177/0x190 [ 7828.084418] _raw_spin_lock+0x21/0x30 [ 7828.084424] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7828.086036] ? do_raw_spin_unlock+0x75/0x190 [ 7828.086050] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7828.086818] ? do_wait_intr+0xf0/0xf0 [ 7828.086830] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7828.088116] mdc_reint+0x2d/0x110 [mdc] [ 7828.088701] mdc_unlink+0x658/0x870 [mdc] [ 7828.088776] lmv_unlink+0x34d/0xbd0 [lmv] [ 7828.088872] ll_unlink+0x24d/0x7f0 [lustre] [ 7828.089053] vfs_unlink+0x202/0x330 [ 7828.089062] do_unlinkat+0x2fa/0x460 [ 7828.089066] __x64_sys_unlink+0x37/0x50 [ 7828.089070] do_syscall_64+0xc1/0x440 [ 7828.089076] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7828.089084] RIP: 0033:0x7f83e5f5c10b [ 7828.089434] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7828.089440] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7828.089448] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7828.089451] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7828.089453] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7828.089456] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7828.089458] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7828.460467] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7828.462960] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7828.465483] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7828.467714] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7828.469752] ? woken_wake_function+0x30/0x30 [ 7828.471460] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7828.473716] kthread+0x1d1/0x200 [ 7828.474820] ? set_kthread_struct+0x70/0x70 [ 7828.476353] ret_from_fork+0x1f/0x30 [ 7832.060031] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 7832.064337] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7832.127788] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7832.139987] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7832.152382] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7832.158687] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7832.164766] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7832.187833] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7832.192830] RAX: 0000000000000cbf RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7832.207070] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7832.217324] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7832.229275] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7832.237810] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7832.247244] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7832.258697] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7832.266886] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7832.276818] Call Trace: [ 7832.280670] [ 7832.283517] ? show_regs.cold.9+0x22/0x2f [ 7832.288587] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7832.292391] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7832.298289] ? watchdog+0x40/0x40 [ 7832.302321] ? hrtimer_interrupt+0x128/0x2c0 [ 7832.305846] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7832.312484] ? apic_timer_interrupt+0xf/0x20 [ 7832.317950] [ 7832.319231] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7832.329372] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7832.336570] do_raw_spin_lock+0x177/0x190 [ 7832.342424] _raw_spin_lock+0x21/0x30 [ 7832.343982] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7832.350903] process_one_work+0x2c8/0x700 [ 7832.352360] worker_thread+0x296/0x6e0 [ 7832.357068] ? rescuer_thread+0x570/0x570 [ 7832.359484] kthread+0x1d1/0x200 [ 7832.365573] ? set_kthread_struct+0x70/0x70 [ 7832.368843] ret_from_fork+0x1f/0x30 [ 7836.059373] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 7836.066846] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7836.112109] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7836.121100] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7836.128553] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7836.136325] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7836.166349] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7836.171640] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7836.174417] RDX: 0000000000007519 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7836.178704] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7836.182100] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7836.186085] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7836.189818] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7836.193374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7836.195833] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7836.199281] Call Trace: [ 7836.200386] [ 7836.201217] ? show_regs.cold.9+0x22/0x2f [ 7836.203136] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7836.205710] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7836.207911] ? watchdog+0x40/0x40 [ 7836.209497] ? hrtimer_interrupt+0x128/0x2c0 [ 7836.211610] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7836.214354] ? apic_timer_interrupt+0xf/0x20 [ 7836.216794] [ 7836.219503] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7836.221590] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7836.230789] do_raw_spin_lock+0x177/0x190 [ 7836.237871] _raw_spin_lock+0x21/0x30 [ 7836.241461] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7836.246627] obd_mod_free+0x46/0x140 [ptlrpc] [ 7836.252600] mdc_commit_open+0x91/0xb0 [mdc] [ 7836.257324] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7836.260667] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7836.264095] after_reply+0xe2c/0x1780 [ptlrpc] [ 7836.267526] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7836.270994] ? __next_timer_interrupt+0x160/0x160 [ 7836.273320] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7836.275693] ? do_wait_intr+0xf0/0xf0 [ 7836.278122] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7836.283376] kthread+0x1d1/0x200 [ 7836.286067] ? set_kthread_struct+0x70/0x70 [ 7836.289718] ret_from_fork+0x1f/0x30 [ 7841.250613] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5328s! [ 7841.259656] Showing busy workqueues and worker pools: [ 7841.265642] workqueue events: flags=0x0 [ 7841.269102] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7841.271590] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7841.295695] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7841.302997] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7841.315242] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7841.321315] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7841.330197] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7841.332264] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7841.342433] workqueue events_long: flags=0x0 [ 7841.344142] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7841.348558] pending: lu_objects_flush [obdclass] [ 7841.350453] workqueue events_unbound: flags=0x2 [ 7841.352930] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7841.355259] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7841.357986] workqueue events_power_efficient: flags=0x80 [ 7841.360505] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7841.365807] pending: neigh_periodic_work, do_cache_clean [ 7841.367884] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7841.376803] pending: neigh_periodic_work, check_lifetime [ 7841.380198] workqueue mm_percpu_wq: flags=0x8 [ 7841.383236] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7841.388362] pending: vmstat_update, lru_add_drain_per_cpu [ 7841.394735] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7841.400834] pending: vmstat_update [ 7841.404652] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7841.411803] pending: lru_add_drain_per_cpu BAR(48) [ 7841.413411] workqueue ipv6_addrconf: flags=0x40008 [ 7841.426391] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7841.429505] pending: addrconf_verify_work [ 7841.431139] workqueue knbd0-recv: flags=0x1a [ 7841.436859] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7841.441655] in-flight: 650:recv_work [ 7841.445582] workqueue ptlrpc_pinger: flags=0x2 [ 7841.457360] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7841.460906] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7841.467015] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5328s workers=4 idle: 36 466 29757 [ 7841.474085] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7831s workers=2 idle: 652 [ 7856.061924] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 7856.068549] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel [ 7856.068630] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7856.068673] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7856.068730] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7856.068735] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7856.068738] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7856.068762] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7856.068765] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7856.097636] ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7856.097661] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7856.097668] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7856.097671] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7856.097686] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7856.097690] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7856.097694] RAX: 0000000000007e89 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7856.097696] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7856.097699] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7856.097701] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7856.097703] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7856.097706] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7856.097708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7856.097711] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7856.097721] Call Trace: [ 7856.097727] [ 7856.097733] ? show_regs.cold.9+0x22/0x2f [ 7856.097741] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7856.097762] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7856.097767] ? watchdog+0x40/0x40 [ 7856.097771] ? hrtimer_interrupt+0x128/0x2c0 [ 7856.097776] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7856.097785] ? apic_timer_interrupt+0xf/0x20 [ 7856.097790] [ 7856.097793] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7856.097797] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7856.097804] do_raw_spin_lock+0x177/0x190 [ 7856.097809] _raw_spin_lock+0x21/0x30 [ 7856.097814] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7856.098234] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7856.098702] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7856.099123] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7856.099330] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7856.100262] ? woken_wake_function+0x30/0x30 [ 7856.100275] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7856.102221] kthread+0x1d1/0x200 [ 7856.102231] ? set_kthread_struct+0x70/0x70 [ 7856.102235] ret_from_fork+0x1f/0x30 [ 7856.354521] RAX: 0000000000007d08 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7856.358728] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7856.365034] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7856.368833] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7856.372547] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7856.376194] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7856.379932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7856.382419] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7856.385692] Call Trace: [ 7856.386959] [ 7856.387895] ? show_regs.cold.9+0x22/0x2f [ 7856.389803] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7856.392582] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7856.397674] ? watchdog+0x40/0x40 [ 7856.399507] ? hrtimer_interrupt+0x128/0x2c0 [ 7856.402000] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7856.404538] ? apic_timer_interrupt+0xf/0x20 [ 7856.406770] [ 7856.407930] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7856.410629] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7856.412327] do_raw_spin_lock+0x177/0x190 [ 7856.413565] _raw_spin_lock+0x21/0x30 [ 7856.415312] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7856.418350] ? do_raw_spin_unlock+0x75/0x190 [ 7856.420505] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7856.424303] ? do_wait_intr+0xf0/0xf0 [ 7856.426844] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7856.433418] mdc_reint+0x2d/0x110 [mdc] [ 7856.438535] mdc_unlink+0x658/0x870 [mdc] [ 7856.443239] lmv_unlink+0x34d/0xbd0 [lmv] [ 7856.447031] ll_unlink+0x24d/0x7f0 [lustre] [ 7856.451587] vfs_unlink+0x202/0x330 [ 7856.459078] do_unlinkat+0x2fa/0x460 [ 7856.467261] __x64_sys_unlink+0x37/0x50 [ 7856.473863] do_syscall_64+0xc1/0x440 [ 7856.479410] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7856.483484] RIP: 0033:0x7f83e5f5c10b [ 7856.486402] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7856.506920] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7856.520197] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7856.528248] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7856.533334] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7856.541916] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7856.555779] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7860.060033] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 7860.063859] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7860.082569] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7860.087858] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7860.091576] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7860.094717] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7860.096653] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7860.104032] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7860.107081] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7860.109755] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7860.112714] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7860.115498] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7860.118509] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7860.121143] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7860.124437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7860.127103] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7860.130014] Call Trace: [ 7860.131043] [ 7860.131772] ? show_regs.cold.9+0x22/0x2f [ 7860.133640] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7860.136136] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7860.138025] ? watchdog+0x40/0x40 [ 7860.139368] ? hrtimer_interrupt+0x128/0x2c0 [ 7860.141342] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7860.143265] ? apic_timer_interrupt+0xf/0x20 [ 7860.144982] [ 7860.145775] ? native_safe_halt+0x2e/0x50 [ 7860.147426] kvm_wait+0xcd/0xe0 [ 7860.148675] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7860.150797] do_raw_spin_lock+0x177/0x190 [ 7860.152463] _raw_spin_lock+0x21/0x30 [ 7860.154015] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7860.156276] process_one_work+0x2c8/0x700 [ 7860.157805] worker_thread+0x296/0x6e0 [ 7860.159695] ? rescuer_thread+0x570/0x570 [ 7860.161361] kthread+0x1d1/0x200 [ 7860.162761] ? set_kthread_struct+0x70/0x70 [ 7860.164357] ret_from_fork+0x1f/0x30 [ 7868.056832] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 7868.061760] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7868.103773] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7868.120359] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7868.135943] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7868.137333] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7868.151678] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7868.164466] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7868.171799] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7868.181290] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7868.186231] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7868.190604] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7868.195352] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7868.199977] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7868.202520] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7868.207520] Call Trace: [ 7868.209080] [ 7868.209853] ? show_regs.cold.9+0x22/0x2f [ 7868.213470] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7868.218570] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7868.228746] ? watchdog+0x40/0x40 [ 7868.230575] ? hrtimer_interrupt+0x128/0x2c0 [ 7868.232751] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7868.235157] ? apic_timer_interrupt+0xf/0x20 [ 7868.238760] [ 7868.239721] ? native_safe_halt+0x2e/0x50 [ 7868.241934] kvm_wait+0xcd/0xe0 [ 7868.245231] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7868.248710] do_raw_spin_lock+0x177/0x190 [ 7868.252402] _raw_spin_lock+0x21/0x30 [ 7868.255264] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7868.262538] obd_mod_free+0x46/0x140 [ptlrpc] [ 7868.269616] mdc_commit_open+0x91/0xb0 [mdc] [ 7868.275806] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7868.284552] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7868.290239] after_reply+0xe2c/0x1780 [ptlrpc] [ 7868.294822] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7868.299786] ? __next_timer_interrupt+0x160/0x160 [ 7868.307686] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7868.313283] ? do_wait_intr+0xf0/0xf0 [ 7868.314594] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7868.320728] kthread+0x1d1/0x200 [ 7868.324605] ? set_kthread_struct+0x70/0x70 [ 7868.326729] ret_from_fork+0x1f/0x30 [ 7871.969065] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5358s! [ 7871.972880] Showing busy workqueues and worker pools: [ 7871.975808] workqueue events: flags=0x0 [ 7871.978175] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7871.982010] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7871.987882] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7871.991276] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7871.995031] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7871.998575] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7872.003049] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7872.006721] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7872.012255] workqueue events_long: flags=0x0 [ 7872.014627] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7872.018757] pending: lu_objects_flush [obdclass] [ 7872.021382] workqueue events_unbound: flags=0x2 [ 7872.023811] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7872.026975] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7872.029864] workqueue events_power_efficient: flags=0x80 [ 7872.033017] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7872.036644] pending: neigh_periodic_work, do_cache_clean [ 7872.039716] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7872.043884] pending: neigh_periodic_work, check_lifetime [ 7872.046853] workqueue mm_percpu_wq: flags=0x8 [ 7872.049307] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7872.052983] pending: vmstat_update, lru_add_drain_per_cpu [ 7872.055965] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7872.059397] pending: vmstat_update [ 7872.061370] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7872.065076] pending: lru_add_drain_per_cpu BAR(48) [ 7872.067308] workqueue ipv6_addrconf: flags=0x40008 [ 7872.071180] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7872.074576] pending: addrconf_verify_work [ 7872.076855] workqueue knbd0-recv: flags=0x1a [ 7872.079470] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7872.083453] in-flight: 650:recv_work [ 7872.085556] workqueue ptlrpc_pinger: flags=0x2 [ 7872.088436] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7872.090898] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7872.093762] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5358s workers=4 idle: 36 466 29757 [ 7872.097374] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7862s workers=2 idle: 652 [ 7884.059074] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 7884.062720] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7884.062919] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7884.062991] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7884.062997] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7884.063000] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7884.063014] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7884.063018] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7884.063023] RAX: 0000000000005f14 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7884.063025] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7884.063027] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7884.063029] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7884.063032] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7884.063035] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7884.063038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7884.063040] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7884.063049] Call Trace: [ 7884.063054] [ 7884.063059] ? show_regs.cold.9+0x22/0x2f [ 7884.063068] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7884.063074] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7884.063078] ? watchdog+0x40/0x40 [ 7884.063082] ? hrtimer_interrupt+0x128/0x2c0 [ 7884.063086] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7884.063094] ? apic_timer_interrupt+0xf/0x20 [ 7884.063099] [ 7884.063102] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7884.063106] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7884.063112] do_raw_spin_lock+0x177/0x190 [ 7884.063116] _raw_spin_lock+0x21/0x30 [ 7884.063120] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7884.063398] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7884.063608] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7884.064563] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7884.064815] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7884.065048] ? woken_wake_function+0x30/0x30 [ 7884.065059] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7884.065277] kthread+0x1d1/0x200 [ 7884.065282] ? set_kthread_struct+0x70/0x70 [ 7884.065285] ret_from_fork+0x1f/0x30 [ 7884.207285] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7884.226233] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7884.231150] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7884.235179] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7884.238312] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7884.247552] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7884.251358] RAX: 0000000000003109 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7884.254500] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7884.257921] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7884.264433] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7884.267664] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7884.271323] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7884.275593] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7884.278864] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7884.282226] Call Trace: [ 7884.283702] [ 7884.284462] ? show_regs.cold.9+0x22/0x2f [ 7884.286587] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7884.289377] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7884.291915] ? watchdog+0x40/0x40 [ 7884.293379] ? hrtimer_interrupt+0x128/0x2c0 [ 7884.296111] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7884.298389] ? apic_timer_interrupt+0xf/0x20 [ 7884.300208] [ 7884.301340] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7884.303892] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7884.306390] do_raw_spin_lock+0x177/0x190 [ 7884.308293] _raw_spin_lock+0x21/0x30 [ 7884.310385] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7884.313490] ? do_raw_spin_unlock+0x75/0x190 [ 7884.315532] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7884.318110] ? do_wait_intr+0xf0/0xf0 [ 7884.319468] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7884.322394] mdc_reint+0x2d/0x110 [mdc] [ 7884.324503] mdc_unlink+0x658/0x870 [mdc] [ 7884.327854] lmv_unlink+0x34d/0xbd0 [lmv] [ 7884.329837] ll_unlink+0x24d/0x7f0 [lustre] [ 7884.331881] vfs_unlink+0x202/0x330 [ 7884.333906] do_unlinkat+0x2fa/0x460 [ 7884.336217] __x64_sys_unlink+0x37/0x50 [ 7884.338331] do_syscall_64+0xc1/0x440 [ 7884.339993] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7884.341878] RIP: 0033:0x7f83e5f5c10b [ 7884.343371] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7884.350958] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7884.353778] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7884.356415] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7884.359155] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7884.361043] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7884.363748] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7888.060012] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 7888.063553] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7888.083320] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7888.088354] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7888.092604] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7888.094863] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7888.097570] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7888.106844] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7888.109973] RAX: 000000000000180a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7888.112899] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7888.116449] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7888.119626] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7888.122896] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7888.126926] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7888.130610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7888.133348] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7888.136028] Call Trace: [ 7888.137261] [ 7888.138147] ? show_regs.cold.9+0x22/0x2f [ 7888.140015] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7888.142107] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7888.144268] ? watchdog+0x40/0x40 [ 7888.145734] ? hrtimer_interrupt+0x128/0x2c0 [ 7888.147715] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7888.150047] ? apic_timer_interrupt+0xf/0x20 [ 7888.152270] [ 7888.153143] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7888.155949] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7888.158511] do_raw_spin_lock+0x177/0x190 [ 7888.160156] _raw_spin_lock+0x21/0x30 [ 7888.162215] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7888.164574] process_one_work+0x2c8/0x700 [ 7888.166491] worker_thread+0x296/0x6e0 [ 7888.168212] ? rescuer_thread+0x570/0x570 [ 7888.170136] kthread+0x1d1/0x200 [ 7888.171787] ? set_kthread_struct+0x70/0x70 [ 7888.174031] ret_from_fork+0x1f/0x30 [ 7900.056273] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 7900.059669] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7900.087124] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7900.092965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7900.098701] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7900.102198] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7900.114330] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7900.120018] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7900.128228] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7900.134766] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7900.138333] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7900.145103] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7900.156305] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7900.164581] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7900.178414] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7900.186301] Call Trace: [ 7900.187098] [ 7900.187786] ? show_regs.cold.9+0x22/0x2f [ 7900.196275] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7900.201836] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7900.203987] ? watchdog+0x40/0x40 [ 7900.205985] ? hrtimer_interrupt+0x128/0x2c0 [ 7900.207632] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7900.210335] ? apic_timer_interrupt+0xf/0x20 [ 7900.215983] [ 7900.216905] ? native_safe_halt+0x2e/0x50 [ 7900.221471] kvm_wait+0xcd/0xe0 [ 7900.222496] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7900.224907] do_raw_spin_lock+0x177/0x190 [ 7900.226460] _raw_spin_lock+0x21/0x30 [ 7900.227462] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7900.228807] obd_mod_free+0x46/0x140 [ptlrpc] [ 7900.231980] mdc_commit_open+0x91/0xb0 [mdc] [ 7900.234260] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7900.236800] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7900.238898] after_reply+0xe2c/0x1780 [ptlrpc] [ 7900.240681] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7900.242247] ? __next_timer_interrupt+0x160/0x160 [ 7900.243964] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7900.245945] ? do_wait_intr+0xf0/0xf0 [ 7900.247348] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7900.249494] kthread+0x1d1/0x200 [ 7900.250893] ? set_kthread_struct+0x70/0x70 [ 7900.252470] ret_from_fork+0x1f/0x30 [ 7902.176146] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5388s! [ 7902.187642] Showing busy workqueues and worker pools: [ 7902.194323] workqueue events: flags=0x0 [ 7902.199774] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7902.208643] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7902.226535] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7902.237638] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7902.246008] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7902.262134] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7902.267010] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7902.277474] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7902.286214] workqueue events_long: flags=0x0 [ 7902.287576] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7902.293669] pending: lu_objects_flush [obdclass] [ 7902.302026] workqueue events_unbound: flags=0x2 [ 7902.319942] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7902.327750] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7902.336167] workqueue events_power_efficient: flags=0x80 [ 7902.340195] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7902.344480] pending: neigh_periodic_work, do_cache_clean [ 7902.352101] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7902.361186] pending: neigh_periodic_work, check_lifetime [ 7902.372088] workqueue mm_percpu_wq: flags=0x8 [ 7902.380921] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7902.383256] pending: vmstat_update, lru_add_drain_per_cpu [ 7902.388012] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7902.401643] pending: vmstat_update [ 7902.405070] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7902.409815] pending: lru_add_drain_per_cpu BAR(48) [ 7902.412443] workqueue ipv6_addrconf: flags=0x40008 [ 7902.417321] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7902.423273] pending: addrconf_verify_work [ 7902.428489] workqueue knbd0-recv: flags=0x1a [ 7902.430982] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7902.439243] in-flight: 650:recv_work [ 7902.440747] workqueue ptlrpc_pinger: flags=0x2 [ 7902.449078] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7902.453391] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7902.457455] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5389s workers=4 idle: 36 466 29757 [ 7902.461859] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7892s workers=2 idle: 652 [ 7912.059584] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 7912.063860] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7912.065848] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7912.065933] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7912.065942] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7912.065944] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7912.066014] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7912.066018] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7912.066022] RAX: 0000000000007984 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7912.066025] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7912.066027] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7912.066030] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7912.066033] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7912.066037] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7912.066041] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7912.066043] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7912.066055] Call Trace: [ 7912.066169] [ 7912.066176] ? show_regs.cold.9+0x22/0x2f [ 7912.066189] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7912.066197] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7912.066210] ? watchdog+0x40/0x40 [ 7912.066215] ? hrtimer_interrupt+0x128/0x2c0 [ 7912.066219] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7912.066236] ? apic_timer_interrupt+0xf/0x20 [ 7912.066241] [ 7912.066244] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7912.066249] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7912.066255] do_raw_spin_lock+0x177/0x190 [ 7912.066260] _raw_spin_lock+0x21/0x30 [ 7912.066264] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7912.066744] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7912.067291] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7912.069174] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7912.069607] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7912.073216] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7912.073296] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7912.073304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7912.073307] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7912.073321] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7912.073326] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7912.073331] RAX: 0000000000007488 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7912.073333] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7912.073335] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7912.073343] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7912.073345] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7912.073349] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7912.073351] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7912.073354] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7912.073361] Call Trace: [ 7912.073367] [ 7912.073373] ? show_regs.cold.9+0x22/0x2f [ 7912.073383] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7912.076904] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7912.077701] ? watchdog+0x40/0x40 [ 7912.077706] ? hrtimer_interrupt+0x128/0x2c0 [ 7912.077716] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7912.077725] ? apic_timer_interrupt+0xf/0x20 [ 7912.077730] [ 7912.077734] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7912.077741] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7912.077746] do_raw_spin_lock+0x177/0x190 [ 7912.077752] _raw_spin_lock+0x21/0x30 [ 7912.077760] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7912.078896] ? do_raw_spin_unlock+0x75/0x190 [ 7912.078902] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7912.079131] ? do_wait_intr+0xf0/0xf0 [ 7912.079140] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7912.080270] mdc_reint+0x2d/0x110 [mdc] [ 7912.080315] mdc_unlink+0x658/0x870 [mdc] [ 7912.080342] lmv_unlink+0x34d/0xbd0 [lmv] [ 7912.080373] ll_unlink+0x24d/0x7f0 [lustre] [ 7912.124508] ? woken_wake_function+0x30/0x30 [ 7912.124529] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7912.124770] kthread+0x1d1/0x200 [ 7912.124775] ? set_kthread_struct+0x70/0x70 [ 7912.124778] ret_from_fork+0x1f/0x30 [ 7912.394828] vfs_unlink+0x202/0x330 [ 7912.397274] do_unlinkat+0x2fa/0x460 [ 7912.399766] __x64_sys_unlink+0x37/0x50 [ 7912.402729] do_syscall_64+0xc1/0x440 [ 7912.404309] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7912.407974] RIP: 0033:0x7f83e5f5c10b [ 7912.410933] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7912.423040] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7912.427038] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7912.431921] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7912.435328] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7912.439077] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7912.444849] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7916.060039] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 7916.062904] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7916.111526] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7916.118005] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7916.123898] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7916.128786] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7916.132477] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7916.144125] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7916.147693] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7916.151547] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7916.154796] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7916.159741] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7916.165049] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7916.168896] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7916.177974] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7916.183914] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7916.192926] Call Trace: [ 7916.195720] [ 7916.196898] ? show_regs.cold.9+0x22/0x2f [ 7916.200552] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7916.202919] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7916.210316] ? watchdog+0x40/0x40 [ 7916.213008] ? hrtimer_interrupt+0x128/0x2c0 [ 7916.217585] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7916.220332] ? apic_timer_interrupt+0xf/0x20 [ 7916.223860] [ 7916.224806] ? native_safe_halt+0x2e/0x50 [ 7916.228533] kvm_wait+0xcd/0xe0 [ 7916.229341] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7916.231230] do_raw_spin_lock+0x177/0x190 [ 7916.233540] _raw_spin_lock+0x21/0x30 [ 7916.235000] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7916.239769] process_one_work+0x2c8/0x700 [ 7916.243597] worker_thread+0x296/0x6e0 [ 7916.245877] ? rescuer_thread+0x570/0x570 [ 7916.248151] kthread+0x1d1/0x200 [ 7916.250680] ? set_kthread_struct+0x70/0x70 [ 7916.252865] ret_from_fork+0x1f/0x30 [ 7928.056016] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 7928.058943] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7928.081543] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7928.085775] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7928.088990] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7928.090828] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7928.097910] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7928.100748] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7928.103749] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7928.106363] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7928.108945] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7928.111494] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7928.114196] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7928.117181] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7928.119521] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7928.122459] Call Trace: [ 7928.123420] [ 7928.124186] ? show_regs.cold.9+0x22/0x2f [ 7928.125729] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7928.127376] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7928.128999] ? watchdog+0x40/0x40 [ 7928.130309] ? hrtimer_interrupt+0x128/0x2c0 [ 7928.132203] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7928.134097] ? apic_timer_interrupt+0xf/0x20 [ 7928.135744] [ 7928.136511] ? native_safe_halt+0x2e/0x50 [ 7928.138055] kvm_wait+0xcd/0xe0 [ 7928.139160] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7928.141241] do_raw_spin_lock+0x177/0x190 [ 7928.142676] _raw_spin_lock+0x21/0x30 [ 7928.143705] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7928.145626] obd_mod_free+0x46/0x140 [ptlrpc] [ 7928.147480] mdc_commit_open+0x91/0xb0 [mdc] [ 7928.149112] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7928.151180] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7928.153346] after_reply+0xe2c/0x1780 [ptlrpc] [ 7928.155502] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7928.157721] ? __next_timer_interrupt+0x160/0x160 [ 7928.159587] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7928.161461] ? do_wait_intr+0xf0/0xf0 [ 7928.162884] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7928.164937] kthread+0x1d1/0x200 [ 7928.166324] ? set_kthread_struct+0x70/0x70 [ 7928.168023] ret_from_fork+0x1f/0x30 [ 7932.896330] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5419s! [ 7932.900524] Showing busy workqueues and worker pools: [ 7932.903219] workqueue events: flags=0x0 [ 7932.904977] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7932.908310] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7932.911644] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7932.914579] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7932.917957] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7932.921240] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7932.924764] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7932.927652] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7932.932346] workqueue events_long: flags=0x0 [ 7932.934059] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7932.937419] pending: lu_objects_flush [obdclass] [ 7932.939928] workqueue events_unbound: flags=0x2 [ 7932.941687] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7932.943601] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7932.946854] workqueue events_power_efficient: flags=0x80 [ 7932.949440] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7932.953445] pending: neigh_periodic_work, do_cache_clean [ 7932.956193] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7932.960312] pending: neigh_periodic_work, check_lifetime [ 7932.963025] workqueue mm_percpu_wq: flags=0x8 [ 7932.965443] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7932.968584] pending: vmstat_update, lru_add_drain_per_cpu [ 7932.971417] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7932.976403] pending: vmstat_update [ 7932.978209] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7932.980916] pending: lru_add_drain_per_cpu BAR(48) [ 7932.982883] workqueue ipv6_addrconf: flags=0x40008 [ 7932.984780] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7932.987843] pending: addrconf_verify_work [ 7932.989837] workqueue knbd0-recv: flags=0x1a [ 7932.991753] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7932.994600] in-flight: 650:recv_work [ 7932.996506] workqueue ptlrpc_pinger: flags=0x2 [ 7932.998626] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7933.001323] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7933.004046] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5419s workers=4 idle: 36 466 29757 [ 7933.008501] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7923s workers=2 idle: 652 [ 7940.059079] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 7940.062044] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7940.063722] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr [ 7940.065860] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7940.065944] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7940.065951] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7940.065954] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7940.065968] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7940.065972] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7940.065976] RAX: 000000000000714c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7940.065979] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7940.065981] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7940.065983] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7940.065985] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7940.065988] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7940.065991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7940.065993] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7940.066001] Call Trace: [ 7940.066005] [ 7940.066010] ? show_regs.cold.9+0x22/0x2f [ 7940.066018] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7940.066024] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7940.066027] ? watchdog+0x40/0x40 [ 7940.066031] ? hrtimer_interrupt+0x128/0x2c0 [ 7940.066035] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7940.066041] ? apic_timer_interrupt+0xf/0x20 [ 7940.066046] [ 7940.066049] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7940.066053] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7940.066057] do_raw_spin_lock+0x177/0x190 [ 7940.066062] _raw_spin_lock+0x21/0x30 [ 7940.066067] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7940.091334] intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7940.091961] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7940.091968] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7940.091971] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7940.091985] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7940.091990] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7940.091997] RAX: 0000000000005040 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7940.092000] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7940.092002] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7940.092005] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7940.092007] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7940.092011] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7940.092016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7940.092019] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7940.092027] Call Trace: [ 7940.092032] [ 7940.092037] ? show_regs.cold.9+0x22/0x2f [ 7940.092050] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7940.092056] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7940.092060] ? watchdog+0x40/0x40 [ 7940.092065] ? hrtimer_interrupt+0x128/0x2c0 [ 7940.092069] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7940.092077] ? apic_timer_interrupt+0xf/0x20 [ 7940.092082] [ 7940.092085] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7940.092096] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7940.092103] do_raw_spin_lock+0x177/0x190 [ 7940.092108] _raw_spin_lock+0x21/0x30 [ 7940.092112] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7940.092444] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7940.092636] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7940.092847] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7940.093792] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7940.094052] ? woken_wake_function+0x30/0x30 [ 7940.094061] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7940.120447] ? do_raw_spin_unlock+0x75/0x190 [ 7940.126861] kthread+0x1d1/0x200 [ 7940.130738] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7940.139476] ? set_kthread_struct+0x70/0x70 [ 7940.139491] ret_from_fork+0x1f/0x30 [ 7940.305749] ? do_wait_intr+0xf0/0xf0 [ 7940.307180] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7940.309740] mdc_reint+0x2d/0x110 [mdc] [ 7940.311246] mdc_unlink+0x658/0x870 [mdc] [ 7940.313506] lmv_unlink+0x34d/0xbd0 [lmv] [ 7940.314690] ll_unlink+0x24d/0x7f0 [lustre] [ 7940.316564] vfs_unlink+0x202/0x330 [ 7940.317672] do_unlinkat+0x2fa/0x460 [ 7940.318807] __x64_sys_unlink+0x37/0x50 [ 7940.320012] do_syscall_64+0xc1/0x440 [ 7940.321086] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7940.322682] RIP: 0033:0x7f83e5f5c10b [ 7940.324072] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7940.329874] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7940.332294] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7940.335297] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7940.337945] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7940.340337] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7940.342942] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7944.061338] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 7944.067192] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7944.100855] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7944.110288] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7944.116430] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7944.123368] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7944.128205] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7944.140471] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7944.145397] RAX: 0000000000007d2d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7944.151175] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7944.155118] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7944.162355] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7944.169299] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7944.179431] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7944.182355] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7944.185216] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7944.188662] Call Trace: [ 7944.190242] [ 7944.191200] ? show_regs.cold.9+0x22/0x2f [ 7944.192880] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7944.195155] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7944.197255] ? watchdog+0x40/0x40 [ 7944.198816] ? hrtimer_interrupt+0x128/0x2c0 [ 7944.201251] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7944.203562] ? apic_timer_interrupt+0xf/0x20 [ 7944.207544] [ 7944.208658] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7944.214261] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7944.217286] do_raw_spin_lock+0x177/0x190 [ 7944.219490] _raw_spin_lock+0x21/0x30 [ 7944.221549] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7944.225173] process_one_work+0x2c8/0x700 [ 7944.227163] worker_thread+0x296/0x6e0 [ 7944.229159] ? rescuer_thread+0x570/0x570 [ 7944.231034] kthread+0x1d1/0x200 [ 7944.233225] ? set_kthread_struct+0x70/0x70 [ 7944.235278] ret_from_fork+0x1f/0x30 [ 7960.056026] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 7960.059760] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7960.099733] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7960.111265] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7960.117069] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7960.118431] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7960.129554] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7960.133591] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 7960.138298] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 7960.143118] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 7960.146483] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 7960.149848] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7960.153310] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 7960.157260] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7960.160881] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 7960.164566] Call Trace: [ 7960.165828] [ 7960.166546] ? show_regs.cold.9+0x22/0x2f [ 7960.168430] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7960.170850] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7960.173332] ? watchdog+0x40/0x40 [ 7960.174983] ? hrtimer_interrupt+0x128/0x2c0 [ 7960.176618] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7960.178353] ? apic_timer_interrupt+0xf/0x20 [ 7960.180745] [ 7960.182331] ? native_safe_halt+0x2e/0x50 [ 7960.185006] kvm_wait+0xcd/0xe0 [ 7960.186548] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7960.188573] do_raw_spin_lock+0x177/0x190 [ 7960.192909] _raw_spin_lock+0x21/0x30 [ 7960.195135] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7960.198415] obd_mod_free+0x46/0x140 [ptlrpc] [ 7960.200834] mdc_commit_open+0x91/0xb0 [mdc] [ 7960.202766] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7960.205429] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7960.208785] after_reply+0xe2c/0x1780 [ptlrpc] [ 7960.212458] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7960.216424] ? __next_timer_interrupt+0x160/0x160 [ 7960.218891] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7960.221781] ? do_wait_intr+0xf0/0xf0 [ 7960.223869] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7960.226845] kthread+0x1d1/0x200 [ 7960.228248] ? set_kthread_struct+0x70/0x70 [ 7960.230337] ret_from_fork+0x1f/0x30 [ 7963.104070] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5449s! [ 7963.107618] Showing busy workqueues and worker pools: [ 7963.109748] workqueue events: flags=0x0 [ 7963.111683] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7963.118659] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7963.128202] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7963.132649] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7963.140823] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7963.150623] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7963.157359] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7963.161877] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7963.167866] workqueue events_long: flags=0x0 [ 7963.169818] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7963.173396] pending: lu_objects_flush [obdclass] [ 7963.176160] workqueue events_unbound: flags=0x2 [ 7963.178562] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7963.181891] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7963.185456] workqueue events_power_efficient: flags=0x80 [ 7963.189139] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7963.191794] pending: neigh_periodic_work, do_cache_clean [ 7963.194839] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7963.200508] pending: neigh_periodic_work, check_lifetime [ 7963.204121] workqueue mm_percpu_wq: flags=0x8 [ 7963.207371] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7963.211758] pending: vmstat_update, lru_add_drain_per_cpu [ 7963.217213] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7963.222136] pending: vmstat_update [ 7963.224528] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7963.228083] pending: lru_add_drain_per_cpu BAR(48) [ 7963.230977] workqueue ipv6_addrconf: flags=0x40008 [ 7963.235953] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7963.242626] pending: addrconf_verify_work [ 7963.248749] workqueue knbd0-recv: flags=0x1a [ 7963.250860] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7963.254291] in-flight: 650:recv_work [ 7963.256390] workqueue ptlrpc_pinger: flags=0x2 [ 7963.258643] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7963.261725] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7963.265821] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5450s workers=4 idle: 36 466 29757 [ 7963.269381] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7953s workers=2 idle: 652 [ 7968.062109] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 7968.064099] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 7968.067851] Modules linked in: [ 7968.071817] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7968.072175] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7968.072183] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7968.072186] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7968.072200] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7968.072204] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7968.072209] RAX: 0000000000006310 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7968.072211] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7968.072214] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7968.072216] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7968.072218] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7968.072221] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7968.072225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7968.072227] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7968.072236] Call Trace: [ 7968.072257] [ 7968.072263] ? show_regs.cold.9+0x22/0x2f [ 7968.072274] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7968.072280] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7968.072284] ? watchdog+0x40/0x40 [ 7968.072288] ? hrtimer_interrupt+0x128/0x2c0 [ 7968.072293] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7968.072300] ? apic_timer_interrupt+0xf/0x20 [ 7968.072304] [ 7968.072307] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7968.072312] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7968.072316] do_raw_spin_lock+0x177/0x190 [ 7968.072321] _raw_spin_lock+0x21/0x30 [ 7968.072327] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7968.072743] ? do_raw_spin_unlock+0x75/0x190 [ 7968.072749] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7968.073092] ? do_wait_intr+0xf0/0xf0 [ 7968.073101] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7968.075197] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common [ 7968.096572] mdc_reint+0x2d/0x110 [mdc] [ 7968.124739] sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7968.124777] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7968.124784] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7968.124788] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7968.124802] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7968.124806] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7968.124811] RAX: 000000000000591e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7968.124813] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7968.124815] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7968.124818] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7968.124821] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7968.124824] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7968.124827] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7968.124830] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7968.124838] Call Trace: [ 7968.124843] [ 7968.124849] ? show_regs.cold.9+0x22/0x2f [ 7968.125073] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7968.125080] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7968.125085] ? watchdog+0x40/0x40 [ 7968.125089] ? hrtimer_interrupt+0x128/0x2c0 [ 7968.125093] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7968.125101] ? apic_timer_interrupt+0xf/0x20 [ 7968.125105] [ 7968.125108] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7968.125113] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7968.125120] do_raw_spin_lock+0x177/0x190 [ 7968.125125] _raw_spin_lock+0x21/0x30 [ 7968.125132] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7968.125620] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7968.125848] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7968.137192] mdc_unlink+0x658/0x870 [mdc] [ 7968.137245] lmv_unlink+0x34d/0xbd0 [lmv] [ 7968.137277] ll_unlink+0x24d/0x7f0 [lustre] [ 7968.138049] vfs_unlink+0x202/0x330 [ 7968.138059] do_unlinkat+0x2fa/0x460 [ 7968.138063] __x64_sys_unlink+0x37/0x50 [ 7968.138067] do_syscall_64+0xc1/0x440 [ 7968.138073] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7968.138082] RIP: 0033:0x7f83e5f5c10b [ 7968.138089] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7968.138093] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7968.138098] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7968.138101] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7968.138103] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7968.138105] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7968.138107] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7968.505254] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7968.507451] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7968.519725] ? woken_wake_function+0x30/0x30 [ 7968.523409] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7968.530509] kthread+0x1d1/0x200 [ 7968.533192] ? set_kthread_struct+0x70/0x70 [ 7968.535621] ret_from_fork+0x1f/0x30 [ 7972.064205] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 7972.071237] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7972.117958] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7972.125073] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7972.129220] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7972.132186] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7972.136047] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7972.148004] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7972.151368] RAX: 000000000000291e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7972.163226] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 7972.175289] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 7972.188004] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 7972.197238] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 7972.203338] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 7972.210786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7972.213149] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 7972.220002] Call Trace: [ 7972.222147] [ 7972.223672] ? show_regs.cold.9+0x22/0x2f [ 7972.224972] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7972.230425] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7972.232025] ? watchdog+0x40/0x40 [ 7972.234436] ? hrtimer_interrupt+0x128/0x2c0 [ 7972.238402] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7972.244580] ? apic_timer_interrupt+0xf/0x20 [ 7972.246126] [ 7972.247136] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7972.250929] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7972.254626] do_raw_spin_lock+0x177/0x190 [ 7972.257237] _raw_spin_lock+0x21/0x30 [ 7972.264035] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7972.270993] process_one_work+0x2c8/0x700 [ 7972.275182] worker_thread+0x296/0x6e0 [ 7972.280596] ? rescuer_thread+0x570/0x570 [ 7972.282482] kthread+0x1d1/0x200 [ 7972.289428] ? set_kthread_struct+0x70/0x70 [ 7972.292675] ret_from_fork+0x1f/0x30 [ 7981.100984] rcu: INFO: rcu_sched self-detected stall on CPU [ 7981.103478] rcu: 0-....: (3827516 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 7981.107664] (t=6000148 jiffies g=262441 q=58920) [ 7981.109696] rcu: rcu_sched kthread starved for 5469337 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 7981.113590] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7981.117915] rcu: RCU grace-period kthread stack dump: [ 7981.119440] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7981.123027] Call Trace: [ 7981.124032] __schedule+0x351/0xcb0 [ 7981.125419] schedule+0xc0/0x180 [ 7981.126640] schedule_timeout+0xb4/0x190 [ 7981.128280] ? do_raw_spin_unlock+0x75/0x190 [ 7981.129681] ? __next_timer_interrupt+0x160/0x160 [ 7981.131706] ? prepare_to_swait_event+0xf3/0x140 [ 7981.133406] rcu_gp_kthread+0x8dd/0x15b0 [ 7981.134740] ? force_qs_rnp+0x2c0/0x2c0 [ 7981.136360] kthread+0x1d1/0x200 [ 7981.137462] ? set_kthread_struct+0x70/0x70 [ 7981.138872] ret_from_fork+0x1f/0x30 [ 7981.140465] rcu: Stack dump where RCU GP kthread last ran: [ 7981.142452] Sending NMI from CPU 0 to CPUs 3: [ 7981.145916] NMI backtrace for cpu 3 [ 7981.145925] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7981.145927] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7981.145929] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7981.145931] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7981.145933] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 [ 7981.145936] RAX: 000000000000258d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7981.145937] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 7981.145939] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7981.145940] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 7981.145941] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 7981.145943] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 7981.145944] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7981.145945] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 7981.145946] Call Trace: [ 7981.145947] [ 7981.145948] ? show_regs.cold.9+0x22/0x2f [ 7981.145949] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7981.145950] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7981.145951] ? nmi_handle+0xbd/0x210 [ 7981.145952] ? default_do_nmi+0x4b/0x1b0 [ 7981.145953] ? do_nmi+0x280/0x380 [ 7981.145954] ? end_repeat_nmi+0x16/0x1f [ 7981.145955] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7981.145957] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7981.145958] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7981.145959] [ 7981.145959] do_raw_spin_lock+0x177/0x190 [ 7981.145960] _raw_spin_lock+0x21/0x30 [ 7981.145961] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7981.145962] ? do_raw_spin_unlock+0x75/0x190 [ 7981.145964] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7981.145965] ? do_wait_intr+0xf0/0xf0 [ 7981.145966] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7981.145967] mdc_reint+0x2d/0x110 [mdc] [ 7981.145968] mdc_unlink+0x658/0x870 [mdc] [ 7981.145969] lmv_unlink+0x34d/0xbd0 [lmv] [ 7981.145970] ll_unlink+0x24d/0x7f0 [lustre] [ 7981.145971] vfs_unlink+0x202/0x330 [ 7981.145972] do_unlinkat+0x2fa/0x460 [ 7981.145973] __x64_sys_unlink+0x37/0x50 [ 7981.145974] do_syscall_64+0xc1/0x440 [ 7981.145975] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7981.145976] RIP: 0033:0x7f83e5f5c10b [ 7981.145978] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 7981.145980] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 7981.145983] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 7981.145985] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 7981.145986] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 7981.145992] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 7981.145994] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 7981.146041] NMI backtrace for cpu 0 [ 7981.254887] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7981.258560] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7981.261742] Call Trace: [ 7981.262378] [ 7981.263084] ? dump_stack+0xbb/0x10e [ 7981.264331] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7981.265957] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7981.267439] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7981.269430] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7981.271415] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7981.273279] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7981.274689] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7981.276642] ? update_process_times+0x6d/0xc0 [ 7981.278116] ? tick_sched_handle.isra.18+0x27/0x90 [ 7981.279725] ? tick_sched_timer+0x53/0xb0 [ 7981.281083] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7981.282540] ? tick_sched_handle.isra.18+0x90/0x90 [ 7981.284216] ? hrtimer_interrupt+0x128/0x2c0 [ 7981.285645] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7981.287279] ? apic_timer_interrupt+0xf/0x20 [ 7981.288703] [ 7981.289463] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7981.291479] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7981.293303] ? do_raw_spin_lock+0x177/0x190 [ 7981.294704] ? _raw_spin_lock+0x21/0x30 [ 7981.295948] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7981.297871] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7981.299656] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7981.301440] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7981.303399] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7981.305594] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7981.307696] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7981.309428] ? __next_timer_interrupt+0x160/0x160 [ 7981.310983] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7981.312989] ? do_wait_intr+0xf0/0xf0 [ 7981.314250] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7981.316167] ? kthread+0x1d1/0x200 [ 7981.317382] ? set_kthread_struct+0x70/0x70 [ 7981.318665] ? ret_from_fork+0x1f/0x30 [ 7993.315100] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5480s! [ 7993.320951] Showing busy workqueues and worker pools: [ 7993.322738] workqueue events: flags=0x0 [ 7993.326241] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7993.330169] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 7993.339220] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7993.346442] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 7993.354250] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7993.359256] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 7993.364459] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7993.371488] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 7993.376735] workqueue events_long: flags=0x0 [ 7993.378560] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7993.381248] pending: lu_objects_flush [obdclass] [ 7993.384209] workqueue events_unbound: flags=0x2 [ 7993.386238] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7993.388584] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7993.395012] workqueue events_power_efficient: flags=0x80 [ 7993.400111] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7993.407424] pending: neigh_periodic_work, do_cache_clean [ 7993.413786] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7993.419212] pending: neigh_periodic_work, check_lifetime [ 7993.423524] workqueue mm_percpu_wq: flags=0x8 [ 7993.425271] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7993.431287] pending: vmstat_update, lru_add_drain_per_cpu [ 7993.435916] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7993.441769] pending: vmstat_update [ 7993.444817] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 7993.450104] pending: lru_add_drain_per_cpu BAR(48) [ 7993.453239] workqueue ipv6_addrconf: flags=0x40008 [ 7993.457166] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7993.463526] pending: addrconf_verify_work [ 7993.470066] workqueue knbd0-recv: flags=0x1a [ 7993.471567] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7993.479238] in-flight: 650:recv_work [ 7993.484082] workqueue ptlrpc_pinger: flags=0x2 [ 7993.488287] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7993.494729] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 7993.502553] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5480s workers=4 idle: 36 466 29757 [ 7993.510499] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7983s workers=2 idle: 652 [ 7996.061760] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 7996.070545] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7996.113713] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7996.124132] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7996.132319] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7996.138457] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7996.155091] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7996.162498] RAX: 000000000000530a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 7996.169918] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 7996.177437] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7996.182148] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 7996.185726] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 7996.188556] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 7996.193020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7996.195562] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 7996.198553] Call Trace: [ 7996.200053] [ 7996.200872] ? show_regs.cold.9+0x22/0x2f [ 7996.202725] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7996.204831] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7996.206780] ? watchdog+0x40/0x40 [ 7996.208415] ? hrtimer_interrupt+0x128/0x2c0 [ 7996.210452] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7996.212661] ? apic_timer_interrupt+0xf/0x20 [ 7996.214735] [ 7996.219059] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7996.224256] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7996.227627] do_raw_spin_lock+0x177/0x190 [ 7996.228828] _raw_spin_lock+0x21/0x30 [ 7996.231812] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7996.234328] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7996.240178] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7996.244039] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7996.249719] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7996.255352] ? woken_wake_function+0x30/0x30 [ 7996.258446] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7996.264224] kthread+0x1d1/0x200 [ 7996.268121] ? set_kthread_struct+0x70/0x70 [ 7996.273285] ret_from_fork+0x1f/0x30 [ 8000.060035] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 8000.068531] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8000.107394] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8000.125983] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8000.139987] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8000.148308] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8000.153393] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8000.168208] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8000.175340] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8000.178653] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8000.181383] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8000.187047] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8000.189947] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8000.199067] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8000.203811] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8000.205989] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8000.208707] Call Trace: [ 8000.210269] [ 8000.212120] ? show_regs.cold.9+0x22/0x2f [ 8000.216815] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8000.218667] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8000.224996] ? watchdog+0x40/0x40 [ 8000.228354] ? hrtimer_interrupt+0x128/0x2c0 [ 8000.230756] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8000.233140] ? apic_timer_interrupt+0xf/0x20 [ 8000.234749] [ 8000.235575] ? native_safe_halt+0x2e/0x50 [ 8000.237786] kvm_wait+0xcd/0xe0 [ 8000.239129] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8000.241713] do_raw_spin_lock+0x177/0x190 [ 8000.243455] _raw_spin_lock+0x21/0x30 [ 8000.244905] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8000.246921] process_one_work+0x2c8/0x700 [ 8000.248377] worker_thread+0x296/0x6e0 [ 8000.249783] ? rescuer_thread+0x570/0x570 [ 8000.251039] kthread+0x1d1/0x200 [ 8000.252185] ? set_kthread_struct+0x70/0x70 [ 8000.258704] ret_from_fork+0x1f/0x30 [ 8008.062012] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 8008.068019] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8008.094641] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8008.101114] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8008.107386] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8008.110979] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8008.122390] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8008.127027] RAX: 0000000000004b6d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8008.135550] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8008.143799] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8008.148720] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8008.151714] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8008.156250] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8008.159926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8008.163061] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8008.167817] Call Trace: [ 8008.168904] [ 8008.170052] ? show_regs.cold.9+0x22/0x2f [ 8008.172410] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8008.174898] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8008.177409] ? watchdog+0x40/0x40 [ 8008.179488] ? hrtimer_interrupt+0x128/0x2c0 [ 8008.182019] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8008.185454] ? apic_timer_interrupt+0xf/0x20 [ 8008.188836] [ 8008.189875] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8008.192629] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8008.195572] do_raw_spin_lock+0x177/0x190 [ 8008.198120] _raw_spin_lock+0x21/0x30 [ 8008.200161] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8008.204033] ? do_raw_spin_unlock+0x75/0x190 [ 8008.206619] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8008.209726] ? do_wait_intr+0xf0/0xf0 [ 8008.211452] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8008.215232] mdc_reint+0x2d/0x110 [mdc] [ 8008.217991] mdc_unlink+0x658/0x870 [mdc] [ 8008.219934] lmv_unlink+0x34d/0xbd0 [lmv] [ 8008.222026] ll_unlink+0x24d/0x7f0 [lustre] [ 8008.224161] vfs_unlink+0x202/0x330 [ 8008.225918] do_unlinkat+0x2fa/0x460 [ 8008.227767] __x64_sys_unlink+0x37/0x50 [ 8008.229477] do_syscall_64+0xc1/0x440 [ 8008.231318] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8008.233754] RIP: 0033:0x7f83e5f5c10b [ 8008.235417] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8008.246006] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8008.256824] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8008.261047] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8008.263438] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8008.267405] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8008.271252] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8020.057771] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 8020.077294] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8020.148246] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8020.165228] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8020.183016] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8020.197111] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8020.237465] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8020.242188] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8020.245165] RDX: 0000000000006b03 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8020.248667] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8020.252884] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8020.259079] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8020.276687] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8020.288633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8020.294980] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8020.299047] Call Trace: [ 8020.299935] [ 8020.310171] ? show_regs.cold.9+0x22/0x2f [ 8020.315129] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8020.327186] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8020.328870] ? watchdog+0x40/0x40 [ 8020.338863] ? hrtimer_interrupt+0x128/0x2c0 [ 8020.348660] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8020.357029] ? apic_timer_interrupt+0xf/0x20 [ 8020.363729] [ 8020.367663] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8020.376771] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8020.378762] do_raw_spin_lock+0x177/0x190 [ 8020.388288] _raw_spin_lock+0x21/0x30 [ 8020.389619] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8020.411662] obd_mod_free+0x46/0x140 [ptlrpc] [ 8020.418746] mdc_commit_open+0x91/0xb0 [mdc] [ 8020.426972] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8020.436551] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8020.448773] after_reply+0xe2c/0x1780 [ptlrpc] [ 8020.450706] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8020.460785] ? __next_timer_interrupt+0x160/0x160 [ 8020.466575] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8020.475003] ? do_wait_intr+0xf0/0xf0 [ 8020.480521] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8020.488624] kthread+0x1d1/0x200 [ 8020.495107] ? set_kthread_struct+0x70/0x70 [ 8020.500437] ret_from_fork+0x1f/0x30 [ 8023.523980] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5510s! [ 8023.527243] Showing busy workqueues and worker pools: [ 8023.529600] workqueue events: flags=0x0 [ 8023.532057] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8023.535376] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8023.540301] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8023.544032] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8023.548209] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8023.552257] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8023.559376] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8023.566697] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8023.580217] workqueue events_long: flags=0x0 [ 8023.588650] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8023.596580] pending: lu_objects_flush [obdclass] [ 8023.601908] workqueue events_unbound: flags=0x2 [ 8023.609756] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8023.620492] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8023.634355] workqueue events_power_efficient: flags=0x80 [ 8023.636816] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8023.650552] pending: neigh_periodic_work, do_cache_clean [ 8023.666737] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8023.678511] pending: neigh_periodic_work, check_lifetime [ 8023.686749] workqueue mm_percpu_wq: flags=0x8 [ 8023.694564] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8023.705716] pending: vmstat_update, lru_add_drain_per_cpu [ 8023.710827] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8023.724611] pending: vmstat_update [ 8023.728678] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8023.737235] pending: lru_add_drain_per_cpu BAR(48) [ 8023.743137] workqueue ipv6_addrconf: flags=0x40008 [ 8023.744865] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8023.763837] pending: addrconf_verify_work [ 8023.772311] workqueue knbd0-recv: flags=0x1a [ 8023.778292] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8023.794729] in-flight: 650:recv_work [ 8023.799682] workqueue ptlrpc_pinger: flags=0x2 [ 8023.803592] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8023.809492] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8023.814484] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5510s workers=4 idle: 36 466 29757 [ 8023.829777] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8013s workers=2 idle: 652 [ 8024.059354] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 8024.064938] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8024.147310] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8024.163106] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8024.174426] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8024.184815] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8024.224473] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8024.236381] RAX: 0000000000006286 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8024.251096] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8024.261707] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8024.267750] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8024.274360] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8024.287655] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8024.295752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8024.301798] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8024.312777] Call Trace: [ 8024.314515] [ 8024.322895] ? show_regs.cold.9+0x22/0x2f [ 8024.326304] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8024.334691] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8024.343466] ? watchdog+0x40/0x40 [ 8024.345795] ? hrtimer_interrupt+0x128/0x2c0 [ 8024.349254] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8024.352777] ? apic_timer_interrupt+0xf/0x20 [ 8024.356583] [ 8024.357669] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8024.362360] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8024.366762] do_raw_spin_lock+0x177/0x190 [ 8024.368757] _raw_spin_lock+0x21/0x30 [ 8024.377722] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8024.388162] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8024.392000] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8024.409476] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8024.421399] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8024.428809] ? woken_wake_function+0x30/0x30 [ 8024.435012] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8024.437405] kthread+0x1d1/0x200 [ 8024.453726] ? set_kthread_struct+0x70/0x70 [ 8024.458859] ret_from_fork+0x1f/0x30 [ 8028.060026] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 8028.062726] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8028.101108] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8028.110726] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8028.117841] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8028.124185] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8028.127413] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8028.134431] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8028.138280] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8028.151354] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8028.158240] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8028.164991] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8028.168467] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8028.171194] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8028.174587] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8028.176726] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8028.179530] Call Trace: [ 8028.180426] [ 8028.181032] ? show_regs.cold.9+0x22/0x2f [ 8028.182683] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8028.184464] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8028.186044] ? watchdog+0x40/0x40 [ 8028.187348] ? hrtimer_interrupt+0x128/0x2c0 [ 8028.188873] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8028.190693] ? apic_timer_interrupt+0xf/0x20 [ 8028.193724] [ 8028.194704] ? native_safe_halt+0x2e/0x50 [ 8028.196429] kvm_wait+0xcd/0xe0 [ 8028.197901] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8028.200575] do_raw_spin_lock+0x177/0x190 [ 8028.202367] _raw_spin_lock+0x21/0x30 [ 8028.204954] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8028.208174] process_one_work+0x2c8/0x700 [ 8028.213042] worker_thread+0x296/0x6e0 [ 8028.215954] ? rescuer_thread+0x570/0x570 [ 8028.217741] kthread+0x1d1/0x200 [ 8028.219880] ? set_kthread_struct+0x70/0x70 [ 8028.222167] ret_from_fork+0x1f/0x30 [ 8036.062023] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 8036.065500] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8036.080494] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8036.084304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8036.087510] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8036.088955] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8036.094747] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8036.096970] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8036.099067] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8036.101296] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8036.103488] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8036.105702] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8036.108323] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8036.110677] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8036.112678] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8036.116356] Call Trace: [ 8036.117061] [ 8036.117668] ? show_regs.cold.9+0x22/0x2f [ 8036.119055] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8036.120536] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8036.122146] ? watchdog+0x40/0x40 [ 8036.123315] ? hrtimer_interrupt+0x128/0x2c0 [ 8036.124513] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8036.126515] ? apic_timer_interrupt+0xf/0x20 [ 8036.128333] [ 8036.128891] ? native_safe_halt+0x2e/0x50 [ 8036.130483] kvm_wait+0xcd/0xe0 [ 8036.131628] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8036.133392] do_raw_spin_lock+0x177/0x190 [ 8036.134823] _raw_spin_lock+0x21/0x30 [ 8036.136174] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8036.138346] ? do_raw_spin_unlock+0x75/0x190 [ 8036.139695] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8036.142085] ? do_wait_intr+0xf0/0xf0 [ 8036.143512] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8036.145627] mdc_reint+0x2d/0x110 [mdc] [ 8036.147309] mdc_unlink+0x658/0x870 [mdc] [ 8036.148715] lmv_unlink+0x34d/0xbd0 [lmv] [ 8036.150457] ll_unlink+0x24d/0x7f0 [lustre] [ 8036.170941] vfs_unlink+0x202/0x330 [ 8036.172611] do_unlinkat+0x2fa/0x460 [ 8036.173801] __x64_sys_unlink+0x37/0x50 [ 8036.175374] do_syscall_64+0xc1/0x440 [ 8036.176618] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8036.178833] RIP: 0033:0x7f83e5f5c10b [ 8036.180149] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8036.186952] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8036.189918] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8036.192919] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8036.195549] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8036.198179] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8036.200785] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8048.061271] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 8048.075742] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8048.163635] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8048.188017] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8048.210281] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8048.219651] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8048.245293] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8048.254625] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8048.265290] RDX: 0000000000007e88 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8048.280779] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8048.298013] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8048.315978] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8048.323649] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8048.334693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8048.342118] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8048.344325] Call Trace: [ 8048.352040] [ 8048.352824] ? show_regs.cold.9+0x22/0x2f [ 8048.354247] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8048.368893] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8048.373294] ? watchdog+0x40/0x40 [ 8048.378897] ? hrtimer_interrupt+0x128/0x2c0 [ 8048.387090] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8048.388935] ? apic_timer_interrupt+0xf/0x20 [ 8048.404829] [ 8048.406060] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8048.411596] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8048.428969] do_raw_spin_lock+0x177/0x190 [ 8048.429992] _raw_spin_lock+0x21/0x30 [ 8048.430841] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8048.438662] obd_mod_free+0x46/0x140 [ptlrpc] [ 8048.444021] mdc_commit_open+0x91/0xb0 [mdc] [ 8048.449407] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8048.459117] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8048.468644] after_reply+0xe2c/0x1780 [ptlrpc] [ 8048.474062] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8048.482034] ? __next_timer_interrupt+0x160/0x160 [ 8048.492312] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8048.500081] ? do_wait_intr+0xf0/0xf0 [ 8048.515149] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8048.517168] kthread+0x1d1/0x200 [ 8048.528191] ? set_kthread_struct+0x70/0x70 [ 8048.532301] ret_from_fork+0x1f/0x30 [ 8052.059018] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 8052.061458] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8052.081617] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8052.085710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8052.089119] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8052.090593] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8052.097135] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8052.099578] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8052.102070] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8052.104596] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8052.107954] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8052.113371] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8052.117394] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8052.120998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8052.123629] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8052.126976] Call Trace: [ 8052.128133] [ 8052.128982] ? show_regs.cold.9+0x22/0x2f [ 8052.130978] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8052.133382] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8052.136121] ? watchdog+0x40/0x40 [ 8052.137551] ? hrtimer_interrupt+0x128/0x2c0 [ 8052.139774] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8052.142163] ? apic_timer_interrupt+0xf/0x20 [ 8052.144362] [ 8052.145480] ? native_safe_halt+0x2e/0x50 [ 8052.147679] kvm_wait+0xcd/0xe0 [ 8052.148950] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8052.151609] do_raw_spin_lock+0x177/0x190 [ 8052.153527] _raw_spin_lock+0x21/0x30 [ 8052.155003] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8052.158552] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8052.162633] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8052.166144] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8052.169256] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8052.174884] ? woken_wake_function+0x30/0x30 [ 8052.177959] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8052.184416] kthread+0x1d1/0x200 [ 8052.187403] ? set_kthread_struct+0x70/0x70 [ 8052.193904] ret_from_fork+0x1f/0x30 [ 8054.241677] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5541s! [ 8054.247293] Showing busy workqueues and worker pools: [ 8054.255555] workqueue events: flags=0x0 [ 8054.262588] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8054.265137] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8054.290683] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8054.298801] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8054.307592] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8054.314420] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8054.333249] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8054.343279] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8054.353250] workqueue events_long: flags=0x0 [ 8054.357049] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8054.363619] pending: lu_objects_flush [obdclass] [ 8054.376325] workqueue events_unbound: flags=0x2 [ 8054.377744] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8054.382224] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8054.393801] workqueue events_power_efficient: flags=0x80 [ 8054.396870] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8054.413812] pending: neigh_periodic_work, do_cache_clean [ 8054.427853] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8054.439224] pending: neigh_periodic_work, check_lifetime [ 8054.441391] workqueue mm_percpu_wq: flags=0x8 [ 8054.453572] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8054.472555] pending: vmstat_update, lru_add_drain_per_cpu [ 8054.486021] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8054.495295] pending: vmstat_update [ 8054.501540] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8054.510251] pending: lru_add_drain_per_cpu BAR(48) [ 8054.513843] workqueue ipv6_addrconf: flags=0x40008 [ 8054.516629] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8054.523851] pending: addrconf_verify_work [ 8054.530053] workqueue knbd0-recv: flags=0x1a [ 8054.543761] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8054.557201] in-flight: 650:recv_work [ 8054.558609] workqueue ptlrpc_pinger: flags=0x2 [ 8054.574177] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8054.591272] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8054.603221] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5541s workers=4 idle: 36 466 29757 [ 8054.618796] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8044s workers=2 idle: 652 [ 8056.060041] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 8056.065948] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8056.100864] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8056.107549] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8056.114097] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8056.119416] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8056.130556] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8056.137551] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8056.141489] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8056.145651] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8056.150268] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8056.152799] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8056.158294] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8056.162431] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8056.167475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8056.170903] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8056.175766] Call Trace: [ 8056.176943] [ 8056.177773] ? show_regs.cold.9+0x22/0x2f [ 8056.182179] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8056.183668] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8056.186600] ? watchdog+0x40/0x40 [ 8056.188428] ? hrtimer_interrupt+0x128/0x2c0 [ 8056.189775] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8056.192843] ? apic_timer_interrupt+0xf/0x20 [ 8056.194420] [ 8056.195440] ? native_safe_halt+0x2e/0x50 [ 8056.199733] kvm_wait+0xcd/0xe0 [ 8056.202445] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8056.206056] do_raw_spin_lock+0x177/0x190 [ 8056.208068] _raw_spin_lock+0x21/0x30 [ 8056.211587] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8056.214498] process_one_work+0x2c8/0x700 [ 8056.217432] worker_thread+0x296/0x6e0 [ 8056.221696] ? rescuer_thread+0x570/0x570 [ 8056.224180] kthread+0x1d1/0x200 [ 8056.225781] ? set_kthread_struct+0x70/0x70 [ 8056.228274] ret_from_fork+0x1f/0x30 [ 8064.062577] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 8064.068363] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8064.105803] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8064.113011] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8064.119301] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8064.121651] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8064.132815] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8064.140173] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8064.144384] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8064.146867] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8064.153900] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8064.156366] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8064.161297] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8064.167253] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8064.174335] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8064.177003] Call Trace: [ 8064.178495] [ 8064.179346] ? show_regs.cold.9+0x22/0x2f [ 8064.180359] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8064.181712] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8064.190425] ? watchdog+0x40/0x40 [ 8064.191446] ? hrtimer_interrupt+0x128/0x2c0 [ 8064.193919] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8064.195616] ? apic_timer_interrupt+0xf/0x20 [ 8064.197416] [ 8064.198403] ? native_safe_halt+0x2e/0x50 [ 8064.201291] kvm_wait+0xcd/0xe0 [ 8064.202815] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8064.204328] do_raw_spin_lock+0x177/0x190 [ 8064.205415] _raw_spin_lock+0x21/0x30 [ 8064.210297] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8064.221171] ? do_raw_spin_unlock+0x75/0x190 [ 8064.222434] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8064.230188] ? do_wait_intr+0xf0/0xf0 [ 8064.231526] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8064.234607] mdc_reint+0x2d/0x110 [mdc] [ 8064.238520] mdc_unlink+0x658/0x870 [mdc] [ 8064.242384] lmv_unlink+0x34d/0xbd0 [lmv] [ 8064.244854] ll_unlink+0x24d/0x7f0 [lustre] [ 8064.247491] vfs_unlink+0x202/0x330 [ 8064.250188] do_unlinkat+0x2fa/0x460 [ 8064.252340] __x64_sys_unlink+0x37/0x50 [ 8064.254772] do_syscall_64+0xc1/0x440 [ 8064.256847] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8064.259508] RIP: 0033:0x7f83e5f5c10b [ 8064.261376] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8064.269988] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8064.274581] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8064.278102] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8064.284608] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8064.290648] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8064.296552] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8080.056031] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 8080.059037] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 8080.059365] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8080.070538] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) [ 8080.088264] [ 8080.088272] ptlrpc_gss(O) [ 8080.094275] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8080.094290] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8080.094293] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8080.094307] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8080.094312] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8080.094321] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8080.094323] RDX: 000000000000597c RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8080.094325] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8080.094327] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8080.094330] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8080.094333] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8080.094335] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8080.094338] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8080.094346] Call Trace: [ 8080.094352] [ 8080.094358] ? show_regs.cold.9+0x22/0x2f [ 8080.094368] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8080.094374] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8080.094384] ? watchdog+0x40/0x40 [ 8080.094388] ? hrtimer_interrupt+0x128/0x2c0 [ 8080.094393] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8080.094401] ? apic_timer_interrupt+0xf/0x20 [ 8080.094406] [ 8080.094409] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8080.094414] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8080.094418] do_raw_spin_lock+0x177/0x190 [ 8080.094423] _raw_spin_lock+0x21/0x30 [ 8080.094430] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8080.094724] obd_mod_free+0x46/0x140 [ptlrpc] [ 8080.094921] mdc_commit_open+0x91/0xb0 [mdc] [ 8080.094962] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8080.095165] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8080.097280] ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul [ 8080.098520] after_reply+0xe2c/0x1780 [ptlrpc] [ 8080.105072] crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log [ 8080.108212] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8080.112397] dm_mod [ 8080.118831] ? __next_timer_interrupt+0x160/0x160 [ 8080.118865] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8080.119149] ? do_wait_intr+0xf0/0xf0 [ 8080.119160] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8080.119372] kthread+0x1d1/0x200 [ 8080.119378] ? set_kthread_struct+0x70/0x70 [ 8080.119381] ret_from_fork+0x1f/0x30 [ 8080.221544] sha512_ssse3 sha512_generic [ 8080.223658] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8080.228722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8080.233615] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8080.237115] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8080.246881] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8080.251045] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8080.254735] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8080.258066] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8080.261686] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8080.266170] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8080.270146] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8080.274864] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8080.276762] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8080.280221] Call Trace: [ 8080.282218] [ 8080.283228] ? show_regs.cold.9+0x22/0x2f [ 8080.286526] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8080.288880] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8080.291835] ? watchdog+0x40/0x40 [ 8080.293690] ? hrtimer_interrupt+0x128/0x2c0 [ 8080.295811] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8080.298848] ? apic_timer_interrupt+0xf/0x20 [ 8080.301227] [ 8080.302214] ? native_safe_halt+0x2e/0x50 [ 8080.304413] kvm_wait+0xcd/0xe0 [ 8080.305991] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8080.308498] do_raw_spin_lock+0x177/0x190 [ 8080.310138] _raw_spin_lock+0x21/0x30 [ 8080.312338] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8080.315335] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8080.318265] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8080.321196] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8080.324188] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8080.327756] ? woken_wake_function+0x30/0x30 [ 8080.329872] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8080.332726] kthread+0x1d1/0x200 [ 8080.333695] ? set_kthread_struct+0x70/0x70 [ 8080.335228] ret_from_fork+0x1f/0x30 [ 8084.060029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 8084.063828] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8084.103208] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8084.112954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8084.120686] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8084.122813] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8084.126397] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8084.148165] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8084.155801] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8084.164648] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8084.173060] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8084.183846] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8084.189217] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8084.193869] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8084.202995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8084.212035] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8084.222968] Call Trace: [ 8084.226614] [ 8084.227404] ? show_regs.cold.9+0x22/0x2f [ 8084.229102] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8084.230602] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8084.238885] ? watchdog+0x40/0x40 [ 8084.242441] ? hrtimer_interrupt+0x128/0x2c0 [ 8084.248258] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8084.253937] ? apic_timer_interrupt+0xf/0x20 [ 8084.258809] [ 8084.259865] ? native_safe_halt+0x2e/0x50 [ 8084.264467] kvm_wait+0xcd/0xe0 [ 8084.266662] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8084.274283] do_raw_spin_lock+0x177/0x190 [ 8084.278450] _raw_spin_lock+0x21/0x30 [ 8084.285217] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8084.286962] process_one_work+0x2c8/0x700 [ 8084.288335] worker_thread+0x296/0x6e0 [ 8084.300219] ? rescuer_thread+0x570/0x570 [ 8084.301607] kthread+0x1d1/0x200 [ 8084.309350] ? set_kthread_struct+0x70/0x70 [ 8084.310761] ret_from_fork+0x1f/0x30 [ 8084.966144] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5571s! [ 8084.985138] Showing busy workqueues and worker pools: [ 8084.996282] workqueue events: flags=0x0 [ 8084.999460] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8085.005034] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8085.014012] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8085.017089] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8085.022241] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8085.029702] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8085.039914] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8085.045532] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8085.051615] workqueue events_long: flags=0x0 [ 8085.055309] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8085.061157] pending: lu_objects_flush [obdclass] [ 8085.064205] workqueue events_unbound: flags=0x2 [ 8085.066966] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8085.074000] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8085.082564] workqueue events_power_efficient: flags=0x80 [ 8085.087152] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8085.095246] pending: neigh_periodic_work, do_cache_clean [ 8085.097302] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8085.122895] pending: neigh_periodic_work, check_lifetime [ 8085.126129] workqueue mm_percpu_wq: flags=0x8 [ 8085.128698] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8085.132657] pending: vmstat_update, lru_add_drain_per_cpu [ 8085.135289] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8085.144681] pending: vmstat_update [ 8085.149225] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8085.154716] pending: lru_add_drain_per_cpu BAR(48) [ 8085.157572] workqueue ipv6_addrconf: flags=0x40008 [ 8085.164242] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8085.174440] pending: addrconf_verify_work [ 8085.181183] workqueue knbd0-recv: flags=0x1a [ 8085.185790] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8085.190643] in-flight: 650:recv_work [ 8085.196194] workqueue ptlrpc_pinger: flags=0x2 [ 8085.199563] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8085.203170] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8085.208470] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5572s workers=4 idle: 36 466 29757 [ 8085.213036] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8075s workers=2 idle: 652 [ 8092.065062] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 8092.073852] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8092.096846] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8092.102103] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8092.106273] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8092.109050] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8092.145433] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8092.154236] RAX: 0000000000006886 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8092.167430] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8092.173826] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8092.184969] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8092.195431] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8092.198973] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8092.202339] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8092.209207] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8092.217352] Call Trace: [ 8092.221100] [ 8092.222791] ? show_regs.cold.9+0x22/0x2f [ 8092.226488] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8092.230336] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8092.237476] ? watchdog+0x40/0x40 [ 8092.240684] ? hrtimer_interrupt+0x128/0x2c0 [ 8092.249022] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8092.257065] ? apic_timer_interrupt+0xf/0x20 [ 8092.263307] [ 8092.266306] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8092.276006] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8092.280697] do_raw_spin_lock+0x177/0x190 [ 8092.282701] _raw_spin_lock+0x21/0x30 [ 8092.286294] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8092.293331] ? do_raw_spin_unlock+0x75/0x190 [ 8092.296103] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8092.307496] ? do_wait_intr+0xf0/0xf0 [ 8092.310611] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8092.315248] mdc_reint+0x2d/0x110 [mdc] [ 8092.321825] mdc_unlink+0x658/0x870 [mdc] [ 8092.323441] lmv_unlink+0x34d/0xbd0 [lmv] [ 8092.331593] ll_unlink+0x24d/0x7f0 [lustre] [ 8092.334264] vfs_unlink+0x202/0x330 [ 8092.337095] do_unlinkat+0x2fa/0x460 [ 8092.338890] __x64_sys_unlink+0x37/0x50 [ 8092.341617] do_syscall_64+0xc1/0x440 [ 8092.346932] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8092.351636] RIP: 0033:0x7f83e5f5c10b [ 8092.355270] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8092.379488] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8092.388750] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8092.396428] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8092.401884] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8092.412299] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8092.424086] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8108.059011] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 8108.073844] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8108.129813] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8108.143196] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8108.153777] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8108.162623] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8108.175349] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8108.179498] RAX: 0000000000007456 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8108.182185] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8108.187149] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8108.195832] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8108.200403] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8108.206981] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8108.211801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8108.213863] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8108.216211] Call Trace: [ 8108.218026] [ 8108.219720] ? show_regs.cold.9+0x22/0x2f [ 8108.222123] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8108.227440] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8108.231566] ? watchdog+0x40/0x40 [ 8108.234155] ? hrtimer_interrupt+0x128/0x2c0 [ 8108.237904] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8108.240012] ? apic_timer_interrupt+0xf/0x20 [ 8108.241726] [ 8108.242378] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8108.250201] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8108.253483] do_raw_spin_lock+0x177/0x190 [ 8108.255425] _raw_spin_lock+0x21/0x30 [ 8108.258850] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8108.267608] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8108.278714] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8108.281367] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8108.287063] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8108.293617] ? woken_wake_function+0x30/0x30 [ 8108.295139] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8108.300330] kthread+0x1d1/0x200 [ 8108.303642] ? set_kthread_struct+0x70/0x70 [ 8108.305510] ret_from_fork+0x1f/0x30 [ 8112.056027] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 8112.059853] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8112.063412] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 8112.116002] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8112.131660] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8112.131744] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8112.131751] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8112.131758] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8112.132358] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8112.132370] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8112.132374] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8112.132379] RAX: 0000000000004187 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8112.132381] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8112.132383] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8112.132385] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8112.132387] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8112.132391] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8112.132394] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8112.132397] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8112.132406] Call Trace: [ 8112.132411] [ 8112.132416] ? show_regs.cold.9+0x22/0x2f [ 8112.132425] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8112.132432] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8112.132436] ? watchdog+0x40/0x40 [ 8112.132441] ? hrtimer_interrupt+0x128/0x2c0 [ 8112.132444] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8112.132454] ? apic_timer_interrupt+0xf/0x20 [ 8112.132458] [ 8112.132460] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8112.132537] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8112.132542] do_raw_spin_lock+0x177/0x190 [ 8112.132546] _raw_spin_lock+0x21/0x30 [ 8112.132552] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8112.132789] process_one_work+0x2c8/0x700 [ 8112.132796] worker_thread+0x296/0x6e0 [ 8112.132801] ? rescuer_thread+0x570/0x570 [ 8112.132805] kthread+0x1d1/0x200 [ 8112.132809] ? set_kthread_struct+0x70/0x70 [ 8112.132813] ret_from_fork+0x1f/0x30 [ 8112.336337] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8112.341048] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8112.343632] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8112.365796] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8112.375325] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8112.380904] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8112.391961] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8112.397354] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8112.412759] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8112.417268] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8112.427722] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8112.431928] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8112.440315] Call Trace: [ 8112.442150] [ 8112.443828] ? show_regs.cold.9+0x22/0x2f [ 8112.447977] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8112.450703] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8112.452139] ? watchdog+0x40/0x40 [ 8112.458322] ? hrtimer_interrupt+0x128/0x2c0 [ 8112.461052] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8112.467964] ? apic_timer_interrupt+0xf/0x20 [ 8112.475635] [ 8112.480078] ? native_safe_halt+0x2e/0x50 [ 8112.485064] kvm_wait+0xcd/0xe0 [ 8112.489319] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8112.494758] do_raw_spin_lock+0x177/0x190 [ 8112.496719] _raw_spin_lock+0x21/0x30 [ 8112.498240] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8112.502643] obd_mod_free+0x46/0x140 [ptlrpc] [ 8112.505906] mdc_commit_open+0x91/0xb0 [mdc] [ 8112.508593] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8112.511657] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8112.516102] after_reply+0xe2c/0x1780 [ptlrpc] [ 8112.520767] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8112.526722] ? __next_timer_interrupt+0x160/0x160 [ 8112.528731] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8112.530944] ? do_wait_intr+0xf0/0xf0 [ 8112.533413] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8112.536836] kthread+0x1d1/0x200 [ 8112.538019] ? set_kthread_struct+0x70/0x70 [ 8112.541709] ret_from_fork+0x1f/0x30 [ 8115.686086] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5602s! [ 8115.694239] Showing busy workqueues and worker pools: [ 8115.697679] workqueue events: flags=0x0 [ 8115.699986] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8115.703064] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8115.708683] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8115.714552] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8115.724882] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8115.732419] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8115.741043] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8115.746009] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8115.754342] workqueue events_long: flags=0x0 [ 8115.756250] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8115.759970] pending: lu_objects_flush [obdclass] [ 8115.762724] workqueue events_unbound: flags=0x2 [ 8115.766054] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8115.769839] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8115.772092] workqueue events_power_efficient: flags=0x80 [ 8115.774030] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8115.779850] pending: neigh_periodic_work, do_cache_clean [ 8115.784733] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8115.788797] pending: neigh_periodic_work, check_lifetime [ 8115.794588] workqueue mm_percpu_wq: flags=0x8 [ 8115.796956] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8115.805892] pending: vmstat_update, lru_add_drain_per_cpu [ 8115.808988] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8115.814024] pending: vmstat_update [ 8115.817353] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8115.819445] pending: lru_add_drain_per_cpu BAR(48) [ 8115.825278] workqueue ipv6_addrconf: flags=0x40008 [ 8115.828413] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8115.831072] pending: addrconf_verify_work [ 8115.832764] workqueue knbd0-recv: flags=0x1a [ 8115.834420] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8115.837133] in-flight: 650:recv_work [ 8115.839254] workqueue ptlrpc_pinger: flags=0x2 [ 8115.842023] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8115.847311] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8115.850837] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5602s workers=4 idle: 36 466 29757 [ 8115.854592] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8105s workers=2 idle: 652 [ 8120.062124] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8120.066863] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8120.094128] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8120.099708] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8120.108302] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8120.116099] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8120.151707] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8120.158525] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8120.164546] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8120.170039] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8120.174356] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8120.178784] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8120.188419] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8120.201627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8120.206786] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8120.215076] Call Trace: [ 8120.218501] [ 8120.221679] ? show_regs.cold.9+0x22/0x2f [ 8120.228347] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8120.231777] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8120.235909] ? watchdog+0x40/0x40 [ 8120.238653] ? hrtimer_interrupt+0x128/0x2c0 [ 8120.241027] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8120.242732] ? apic_timer_interrupt+0xf/0x20 [ 8120.245652] [ 8120.246670] ? native_safe_halt+0x2e/0x50 [ 8120.252412] kvm_wait+0xcd/0xe0 [ 8120.256037] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8120.261649] do_raw_spin_lock+0x177/0x190 [ 8120.272128] _raw_spin_lock+0x21/0x30 [ 8120.277505] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8120.286860] ? do_raw_spin_unlock+0x75/0x190 [ 8120.288076] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8120.293094] ? do_wait_intr+0xf0/0xf0 [ 8120.297991] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8120.305836] mdc_reint+0x2d/0x110 [mdc] [ 8120.313243] mdc_unlink+0x658/0x870 [mdc] [ 8120.316358] lmv_unlink+0x34d/0xbd0 [lmv] [ 8120.318292] ll_unlink+0x24d/0x7f0 [lustre] [ 8120.319950] vfs_unlink+0x202/0x330 [ 8120.326610] do_unlinkat+0x2fa/0x460 [ 8120.329132] __x64_sys_unlink+0x37/0x50 [ 8120.331267] do_syscall_64+0xc1/0x440 [ 8120.334662] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8120.338693] RIP: 0033:0x7f83e5f5c10b [ 8120.344732] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8120.354158] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8120.356892] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8120.359641] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8120.362610] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8120.365219] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8120.368213] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8136.059012] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 8136.063924] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8136.086808] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8136.092685] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8136.096683] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8136.101813] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8136.115057] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8136.119560] RAX: 000000000000738a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8136.122534] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8136.125602] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8136.129220] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8136.132402] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8136.135551] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8136.138920] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8136.142044] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8136.145195] Call Trace: [ 8136.146798] [ 8136.147683] ? show_regs.cold.9+0x22/0x2f [ 8136.150160] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8136.152522] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8136.154798] ? watchdog+0x40/0x40 [ 8136.156778] ? hrtimer_interrupt+0x128/0x2c0 [ 8136.158928] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8136.161545] ? apic_timer_interrupt+0xf/0x20 [ 8136.164045] [ 8136.164965] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8136.167888] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8136.170647] do_raw_spin_lock+0x177/0x190 [ 8136.173142] _raw_spin_lock+0x21/0x30 [ 8136.174550] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8136.177794] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8136.181482] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8136.184473] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8136.187121] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8136.189791] ? woken_wake_function+0x30/0x30 [ 8136.192290] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8136.196058] kthread+0x1d1/0x200 [ 8136.197698] ? set_kthread_struct+0x70/0x70 [ 8136.200042] ret_from_fork+0x1f/0x30 [ 8140.056258] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 8140.062034] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 8140.063997] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8140.064077] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8140.064086] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8140.064089] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8140.064104] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8140.064109] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8140.064113] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8140.064116] RDX: 0000000000003ad9 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8140.064118] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8140.064121] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8140.064123] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8140.064126] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8140.064129] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8140.064132] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8140.064141] Call Trace: [ 8140.064147] [ 8140.064153] ? show_regs.cold.9+0x22/0x2f [ 8140.064162] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8140.064168] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8140.064174] ? watchdog+0x40/0x40 [ 8140.064179] ? hrtimer_interrupt+0x128/0x2c0 [ 8140.064184] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8140.064194] ? apic_timer_interrupt+0xf/0x20 [ 8140.064199] [ 8140.064202] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8140.064207] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8140.064212] do_raw_spin_lock+0x177/0x190 [ 8140.064218] _raw_spin_lock+0x21/0x30 [ 8140.064223] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8140.070505] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8140.070589] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8140.070596] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8140.070603] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8140.093481] obd_mod_free+0x46/0x140 [ptlrpc] [ 8140.097399] [ 8140.099738] mdc_commit_open+0x91/0xb0 [mdc] [ 8140.101495] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8140.101513] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8140.101517] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8140.101522] RAX: 0000000000005c0b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8140.101524] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8140.101526] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8140.101527] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8140.101529] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8140.101532] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8140.101534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8140.101536] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8140.101543] Call Trace: [ 8140.101547] [ 8140.101550] ? show_regs.cold.9+0x22/0x2f [ 8140.101558] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8140.101563] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8140.101567] ? watchdog+0x40/0x40 [ 8140.101571] ? hrtimer_interrupt+0x128/0x2c0 [ 8140.101574] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8140.101581] ? apic_timer_interrupt+0xf/0x20 [ 8140.101585] [ 8140.101587] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8140.101591] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8140.101597] do_raw_spin_lock+0x177/0x190 [ 8140.101600] _raw_spin_lock+0x21/0x30 [ 8140.101604] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8140.101863] process_one_work+0x2c8/0x700 [ 8140.101869] worker_thread+0x296/0x6e0 [ 8140.101874] ? rescuer_thread+0x570/0x570 [ 8140.107375] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8140.110175] kthread+0x1d1/0x200 [ 8140.112568] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8140.112841] after_reply+0xe2c/0x1780 [ptlrpc] [ 8140.113116] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8140.113387] ? __next_timer_interrupt+0x160/0x160 [ 8140.113397] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8140.113605] ? do_wait_intr+0xf0/0xf0 [ 8140.113617] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8140.113817] kthread+0x1d1/0x200 [ 8140.113823] ? set_kthread_struct+0x70/0x70 [ 8140.113827] ret_from_fork+0x1f/0x30 [ 8140.421294] ? set_kthread_struct+0x70/0x70 [ 8140.427409] ret_from_fork+0x1f/0x30 [ 8145.890042] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5632s! [ 8145.897359] Showing busy workqueues and worker pools: [ 8145.899079] workqueue events: flags=0x0 [ 8145.902614] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8145.907901] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8145.913039] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8145.916766] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8145.923371] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8145.929522] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8145.936056] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8145.942574] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8145.947771] workqueue events_long: flags=0x0 [ 8145.951255] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8145.962571] pending: lu_objects_flush [obdclass] [ 8145.977597] workqueue events_unbound: flags=0x2 [ 8145.986488] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8145.989045] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8145.995678] workqueue events_power_efficient: flags=0x80 [ 8146.004516] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8146.007842] pending: neigh_periodic_work, do_cache_clean [ 8146.010685] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8146.015435] pending: neigh_periodic_work, check_lifetime [ 8146.018502] workqueue mm_percpu_wq: flags=0x8 [ 8146.020733] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8146.030403] pending: vmstat_update, lru_add_drain_per_cpu [ 8146.035397] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8146.039930] pending: vmstat_update [ 8146.043065] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8146.047132] pending: lru_add_drain_per_cpu BAR(48) [ 8146.049515] workqueue ipv6_addrconf: flags=0x40008 [ 8146.051465] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8146.061688] pending: addrconf_verify_work [ 8146.065399] workqueue knbd0-recv: flags=0x1a [ 8146.068909] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8146.073235] in-flight: 650:recv_work [ 8146.080090] workqueue ptlrpc_pinger: flags=0x2 [ 8146.085477] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8146.087838] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8146.109061] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5632s workers=4 idle: 36 466 29757 [ 8146.116224] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8136s workers=2 idle: 652 [ 8148.062012] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8148.066743] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8148.120982] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8148.143582] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8148.150009] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30a/0x6d0 [ 8148.158410] Code: 03 00 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 [ 8148.182548] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8148.193432] RAX: 0000000000006596 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8148.201469] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8148.205658] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8148.214420] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8148.220961] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8148.228734] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8148.239315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8148.246262] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8148.250229] Call Trace: [ 8148.250811] [ 8148.251328] ? show_regs.cold.9+0x22/0x2f [ 8148.256007] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8148.257734] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8148.261700] ? watchdog+0x40/0x40 [ 8148.272355] ? hrtimer_interrupt+0x128/0x2c0 [ 8148.279007] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8148.286708] ? apic_timer_interrupt+0xf/0x20 [ 8148.288258] [ 8148.292901] ? __pv_queued_spin_lock_slowpath+0x30a/0x6d0 [ 8148.298186] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8148.303531] do_raw_spin_lock+0x177/0x190 [ 8148.306923] _raw_spin_lock+0x21/0x30 [ 8148.309322] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8148.312241] ? do_raw_spin_unlock+0x75/0x190 [ 8148.314558] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8148.321036] ? do_wait_intr+0xf0/0xf0 [ 8148.322567] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8148.325536] mdc_reint+0x2d/0x110 [mdc] [ 8148.329394] mdc_unlink+0x658/0x870 [mdc] [ 8148.333397] lmv_unlink+0x34d/0xbd0 [lmv] [ 8148.338220] ll_unlink+0x24d/0x7f0 [lustre] [ 8148.340279] vfs_unlink+0x202/0x330 [ 8148.345469] do_unlinkat+0x2fa/0x460 [ 8148.348885] __x64_sys_unlink+0x37/0x50 [ 8148.353707] do_syscall_64+0xc1/0x440 [ 8148.355113] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8148.357835] RIP: 0033:0x7f83e5f5c10b [ 8148.358759] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8148.370474] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8148.373769] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8148.377487] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8148.382897] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8148.386750] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8148.388770] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8161.105569] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 8161.111801] rcu: 0-....: (3942984 ticks this GP) idle=30a/1/0x4000000000000000 softirq=128519/128519 fqs=126395 [ 8161.125856] (detected by 1, t=6180146 jiffies, g=262441, q=58983) [ 8161.135703] Sending NMI from CPU 1 to CPUs 0: [ 8161.144266] NMI backtrace for cpu 0 [ 8161.144401] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8161.144406] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8161.144407] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8161.144409] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8161.144412] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 [ 8161.144414] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8161.144416] RDX: 00000000000030ad RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8161.144456] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8161.144458] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8161.144459] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8161.144460] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8161.144462] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8161.144464] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8161.144465] Call Trace: [ 8161.144465] [ 8161.144466] ? show_regs.cold.9+0x22/0x2f [ 8161.144467] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8161.144469] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8161.144470] ? nmi_handle+0xbd/0x210 [ 8161.144471] ? default_do_nmi+0x4b/0x1b0 [ 8161.144472] ? do_nmi+0x280/0x380 [ 8161.144473] ? end_repeat_nmi+0x16/0x1f [ 8161.144474] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8161.144475] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8161.144476] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8161.144477] [ 8161.144478] do_raw_spin_lock+0x177/0x190 [ 8161.144479] _raw_spin_lock+0x21/0x30 [ 8161.144480] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8161.144481] obd_mod_free+0x46/0x140 [ptlrpc] [ 8161.144483] mdc_commit_open+0x91/0xb0 [mdc] [ 8161.144484] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8161.144485] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8161.144486] after_reply+0xe2c/0x1780 [ptlrpc] [ 8161.144487] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8161.144488] ? __next_timer_interrupt+0x160/0x160 [ 8161.144489] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8161.144490] ? do_wait_intr+0xf0/0xf0 [ 8161.144492] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8161.144493] kthread+0x1d1/0x200 [ 8161.144494] ? set_kthread_struct+0x70/0x70 [ 8161.144495] ret_from_fork+0x1f/0x30 [ 8161.144660] rcu: rcu_sched kthread starved for 5649333 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 8161.427706] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8161.445382] rcu: RCU grace-period kthread stack dump: [ 8161.447228] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8161.460368] Call Trace: [ 8161.462345] __schedule+0x351/0xcb0 [ 8161.467589] schedule+0xc0/0x180 [ 8161.473215] schedule_timeout+0xb4/0x190 [ 8161.474664] ? do_raw_spin_unlock+0x75/0x190 [ 8161.477353] ? __next_timer_interrupt+0x160/0x160 [ 8161.483254] ? prepare_to_swait_event+0xf3/0x140 [ 8161.488957] rcu_gp_kthread+0x8dd/0x15b0 [ 8161.490406] ? force_qs_rnp+0x2c0/0x2c0 [ 8161.496166] kthread+0x1d1/0x200 [ 8161.497357] ? set_kthread_struct+0x70/0x70 [ 8161.504556] ret_from_fork+0x1f/0x30 [ 8161.511641] rcu: Stack dump where RCU GP kthread last ran: [ 8161.518392] Sending NMI from CPU 1 to CPUs 2: [ 8161.527436] NMI backtrace for cpu 2 [ 8161.527444] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8161.527446] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8161.527448] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8161.527451] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8161.527453] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8161.527455] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 [ 8161.527457] RAX: 0000000000007d2e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8161.527458] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8161.529499] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8161.529509] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8161.529510] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8161.529512] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8161.529513] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8161.529514] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8161.529515] Call Trace: [ 8161.529516] [ 8161.529517] ? show_regs.cold.9+0x22/0x2f [ 8161.529518] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8161.529519] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8161.529520] ? nmi_handle+0xbd/0x210 [ 8161.529521] ? default_do_nmi+0x4b/0x1b0 [ 8161.529522] ? do_nmi+0x280/0x380 [ 8161.529523] ? end_repeat_nmi+0x16/0x1f [ 8161.529524] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8161.529525] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8161.529526] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8161.529527] [ 8161.529528] do_raw_spin_lock+0x177/0x190 [ 8161.529530] _raw_spin_lock+0x21/0x30 [ 8161.529531] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8161.529532] process_one_work+0x2c8/0x700 [ 8161.529533] worker_thread+0x296/0x6e0 [ 8161.529534] ? rescuer_thread+0x570/0x570 [ 8161.529535] kthread+0x1d1/0x200 [ 8161.529536] ? set_kthread_struct+0x70/0x70 [ 8161.529537] ret_from_fork+0x1f/0x30 [ 8176.066672] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8176.075771] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8176.136156] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8176.155139] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8176.164139] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8176.170863] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8176.192322] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8176.197583] RAX: 0000000000001a2b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8176.205649] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8176.209698] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8176.218296] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8176.224115] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8176.227393] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8176.231848] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8176.235595] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8176.239564] Call Trace: [ 8176.240442] [ 8176.243005] ? show_regs.cold.9+0x22/0x2f [ 8176.247576] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8176.251392] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8176.269430] ? watchdog+0x40/0x40 [ 8176.273713] ? hrtimer_interrupt+0x128/0x2c0 [ 8176.279351] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8176.284036] ? apic_timer_interrupt+0xf/0x20 [ 8176.290742] [ 8176.292018] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8176.297888] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8176.305348] do_raw_spin_lock+0x177/0x190 [ 8176.310574] _raw_spin_lock+0x21/0x30 [ 8176.313595] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8176.319848] ? do_raw_spin_unlock+0x75/0x190 [ 8176.328528] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8176.335707] ? do_wait_intr+0xf0/0xf0 [ 8176.337623] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8176.345903] mdc_reint+0x2d/0x110 [mdc] [ 8176.350478] mdc_unlink+0x658/0x870 [mdc] [ 8176.352210] lmv_unlink+0x34d/0xbd0 [lmv] [ 8176.353579] ll_unlink+0x24d/0x7f0 [lustre] [ 8176.357819] vfs_unlink+0x202/0x330 [ 8176.364343] do_unlinkat+0x2fa/0x460 [ 8176.365794] __x64_sys_unlink+0x37/0x50 [ 8176.372853] do_syscall_64+0xc1/0x440 [ 8176.376358] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8176.386557] RIP: 0033:0x7f83e5f5c10b [ 8176.389265] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8176.413836] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8176.431290] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8176.446252] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8176.453363] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8176.459700] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8176.470937] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8176.619064] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5663s! [ 8176.631665] Showing busy workqueues and worker pools: [ 8176.643620] workqueue events: flags=0x0 [ 8176.645631] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8176.663051] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8176.671780] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8176.684709] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8176.695400] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8176.707824] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8176.719397] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8176.735756] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8176.746391] workqueue events_long: flags=0x0 [ 8176.751466] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8176.755692] pending: lu_objects_flush [obdclass] [ 8176.761992] workqueue events_unbound: flags=0x2 [ 8176.764761] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8176.776137] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8176.790536] workqueue events_power_efficient: flags=0x80 [ 8176.804091] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8176.811065] pending: neigh_periodic_work, do_cache_clean [ 8176.818417] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8176.822149] pending: neigh_periodic_work, check_lifetime [ 8176.825284] workqueue mm_percpu_wq: flags=0x8 [ 8176.828392] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8176.834414] pending: vmstat_update, lru_add_drain_per_cpu [ 8176.838423] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8176.852844] pending: vmstat_update [ 8176.860420] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8176.871049] pending: lru_add_drain_per_cpu BAR(48) [ 8176.880437] workqueue ipv6_addrconf: flags=0x40008 [ 8176.887627] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8176.892699] pending: addrconf_verify_work [ 8176.900384] workqueue knbd0-recv: flags=0x1a [ 8176.904161] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8176.917048] in-flight: 650:recv_work [ 8176.918412] workqueue ptlrpc_pinger: flags=0x2 [ 8176.937132] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8176.952360] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8176.958773] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5663s workers=4 idle: 36 466 29757 [ 8176.969444] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8167s workers=2 idle: 652 [ 8188.059034] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 8188.061965] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8188.062381] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 8188.079278] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8188.079294] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8188.079297] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8188.079309] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8188.079313] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8188.079318] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8188.079321] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8188.079323] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8188.079325] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8188.079328] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8188.079331] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8188.079334] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8188.079336] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8188.079343] Call Trace: [ 8188.079348] [ 8188.079353] ? show_regs.cold.9+0x22/0x2f [ 8188.079362] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8188.079368] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8188.079373] ? watchdog+0x40/0x40 [ 8188.079376] ? hrtimer_interrupt+0x128/0x2c0 [ 8188.079380] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8188.079386] ? apic_timer_interrupt+0xf/0x20 [ 8188.079533] [ 8188.079537] ? native_safe_halt+0x2e/0x50 [ 8188.079541] kvm_wait+0xcd/0xe0 [ 8188.079549] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8188.079560] do_raw_spin_lock+0x177/0x190 [ 8188.079569] _raw_spin_lock+0x21/0x30 [ 8188.079573] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8188.080214] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8188.080549] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8188.080974] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8188.081185] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8188.084024] Modules linked in: lustre(O) osp(O) [ 8188.087986] ? woken_wake_function+0x30/0x30 [ 8188.088009] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8188.088317] kthread+0x1d1/0x200 [ 8188.088324] ? set_kthread_struct+0x70/0x70 [ 8188.088327] ret_from_fork+0x1f/0x30 [ 8188.247500] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8188.278635] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8188.286400] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8188.290509] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8188.294848] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8188.297945] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8188.312676] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8188.316573] RAX: 0000000000007a7a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8188.322781] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8188.326165] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8188.330111] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8188.335472] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8188.340149] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8188.347876] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8188.351205] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8188.354548] Call Trace: [ 8188.357640] [ 8188.358858] ? show_regs.cold.9+0x22/0x2f [ 8188.361002] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8188.363096] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8188.365312] ? watchdog+0x40/0x40 [ 8188.366904] ? hrtimer_interrupt+0x128/0x2c0 [ 8188.368399] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8188.370223] ? apic_timer_interrupt+0xf/0x20 [ 8188.372289] [ 8188.373458] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8188.376116] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8188.379178] do_raw_spin_lock+0x177/0x190 [ 8188.380786] _raw_spin_lock+0x21/0x30 [ 8188.382384] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8188.385108] process_one_work+0x2c8/0x700 [ 8188.387266] worker_thread+0x296/0x6e0 [ 8188.389360] ? rescuer_thread+0x570/0x570 [ 8188.391335] kthread+0x1d1/0x200 [ 8188.393079] ? set_kthread_struct+0x70/0x70 [ 8188.394682] ret_from_fork+0x1f/0x30 [ 8204.056337] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 8204.063488] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8204.066319] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8204.087438] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8204.087537] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8204.087545] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8204.087561] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8204.087565] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8204.087571] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8204.087574] RDX: 0000000000000c71 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8204.087576] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8204.087579] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8204.087581] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8204.087585] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8204.087588] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8204.087590] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8204.087599] Call Trace: [ 8204.087606] [ 8204.087617] ? show_regs.cold.9+0x22/0x2f [ 8204.087628] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8204.087636] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8204.087640] ? watchdog+0x40/0x40 [ 8204.087645] ? hrtimer_interrupt+0x128/0x2c0 [ 8204.087649] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8204.087659] ? apic_timer_interrupt+0xf/0x20 [ 8204.087664] [ 8204.087667] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8204.087672] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8204.087676] do_raw_spin_lock+0x177/0x190 [ 8204.087687] _raw_spin_lock+0x21/0x30 [ 8204.087694] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8204.088193] obd_mod_free+0x46/0x140 [ptlrpc] [ 8204.101031] Modules linked in: lustre(O) osp(O) ofd(O) [ 8204.106677] mdc_commit_open+0x91/0xb0 [mdc] [ 8204.106736] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8204.107336] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8204.125264] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs [ 8204.127694] after_reply+0xe2c/0x1780 [ptlrpc] [ 8204.128094] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8204.128337] ? __next_timer_interrupt+0x160/0x160 [ 8204.128348] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8204.128733] ? do_wait_intr+0xf0/0xf0 [ 8204.128748] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8204.128966] kthread+0x1d1/0x200 [ 8204.128974] ? set_kthread_struct+0x70/0x70 [ 8204.128978] ret_from_fork+0x1f/0x30 [ 8204.346011] ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8204.357120] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8204.365307] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8204.370868] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8204.374551] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8204.397151] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8204.403924] RAX: 000000000000262a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8204.409143] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8204.412923] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8204.419730] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8204.424556] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8204.431939] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8204.437693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8204.439928] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8204.442953] Call Trace: [ 8204.444016] [ 8204.444931] ? show_regs.cold.9+0x22/0x2f [ 8204.453932] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8204.456272] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8204.458319] ? watchdog+0x40/0x40 [ 8204.462460] ? hrtimer_interrupt+0x128/0x2c0 [ 8204.467236] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8204.470914] ? apic_timer_interrupt+0xf/0x20 [ 8204.475807] [ 8204.476550] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8204.479663] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8204.483086] do_raw_spin_lock+0x177/0x190 [ 8204.485180] _raw_spin_lock+0x21/0x30 [ 8204.486997] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8204.489931] ? do_raw_spin_unlock+0x75/0x190 [ 8204.496736] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8204.499359] ? do_wait_intr+0xf0/0xf0 [ 8204.501143] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8204.505954] mdc_reint+0x2d/0x110 [mdc] [ 8204.509952] mdc_unlink+0x658/0x870 [mdc] [ 8204.514235] lmv_unlink+0x34d/0xbd0 [lmv] [ 8204.516028] ll_unlink+0x24d/0x7f0 [lustre] [ 8204.518007] vfs_unlink+0x202/0x330 [ 8204.519508] do_unlinkat+0x2fa/0x460 [ 8204.521478] __x64_sys_unlink+0x37/0x50 [ 8204.522953] do_syscall_64+0xc1/0x440 [ 8204.524308] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8204.526822] RIP: 0033:0x7f83e5f5c10b [ 8204.529663] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8204.544463] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8204.550656] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8204.554389] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8204.558201] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8204.562236] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8204.566279] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8207.330065] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5694s! [ 8207.334068] Showing busy workqueues and worker pools: [ 8207.335416] workqueue events: flags=0x0 [ 8207.336207] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8207.343632] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8207.351117] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8207.352983] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8207.361528] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8207.369055] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8207.376480] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8207.381184] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8207.392514] workqueue events_long: flags=0x0 [ 8207.396818] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8207.400778] pending: lu_objects_flush [obdclass] [ 8207.406126] workqueue events_unbound: flags=0x2 [ 8207.407896] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8207.412807] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8207.418952] workqueue events_power_efficient: flags=0x80 [ 8207.421148] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8207.427959] pending: neigh_periodic_work, do_cache_clean [ 8207.430980] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8207.438156] pending: neigh_periodic_work, check_lifetime [ 8207.445174] workqueue mm_percpu_wq: flags=0x8 [ 8207.449186] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8207.453315] pending: vmstat_update, lru_add_drain_per_cpu [ 8207.455950] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8207.458442] pending: vmstat_update [ 8207.459676] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8207.464387] pending: lru_add_drain_per_cpu BAR(48) [ 8207.468172] workqueue ipv6_addrconf: flags=0x40008 [ 8207.471796] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8207.473420] pending: addrconf_verify_work [ 8207.474433] workqueue knbd0-recv: flags=0x1a [ 8207.475299] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8207.482243] in-flight: 650:recv_work [ 8207.487358] workqueue ptlrpc_pinger: flags=0x2 [ 8207.490826] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8207.496140] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8207.500290] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5694s workers=4 idle: 36 466 29757 [ 8207.508599] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8197s workers=2 idle: 652 [ 8216.059619] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 8216.060307] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 8216.065910] Modules linked in: [ 8216.069745] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8216.069823] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8216.069831] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8216.069838] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8216.076869] lustre(O) [ 8216.098694] [ 8216.098705] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8216.098721] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8216.098725] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8216.098730] RAX: 0000000000001f48 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8216.098733] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8216.098735] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8216.098738] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8216.098740] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8216.098743] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8216.098746] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8216.098748] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8216.098756] Call Trace: [ 8216.098762] [ 8216.098768] ? show_regs.cold.9+0x22/0x2f [ 8216.098778] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8216.098784] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8216.098788] ? watchdog+0x40/0x40 [ 8216.098793] ? hrtimer_interrupt+0x128/0x2c0 [ 8216.098798] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8216.098807] ? apic_timer_interrupt+0xf/0x20 [ 8216.098812] [ 8216.098814] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8216.098819] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8216.098827] do_raw_spin_lock+0x177/0x190 [ 8216.098832] _raw_spin_lock+0x21/0x30 [ 8216.098837] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8216.115892] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8216.115966] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8216.115972] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8216.115975] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8216.115989] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8216.115993] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8216.115997] RAX: 00000000000000fa RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8216.115999] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8216.116001] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8216.116003] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8216.116005] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8216.116009] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8216.116012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8216.116014] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8216.116023] Call Trace: [ 8216.116030] [ 8216.116035] ? show_regs.cold.9+0x22/0x2f [ 8216.116045] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8216.116052] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8216.116056] ? watchdog+0x40/0x40 [ 8216.116061] ? hrtimer_interrupt+0x128/0x2c0 [ 8216.116065] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8216.116075] ? apic_timer_interrupt+0xf/0x20 [ 8216.116080] [ 8216.116083] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8216.116088] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8216.116094] do_raw_spin_lock+0x177/0x190 [ 8216.116099] _raw_spin_lock+0x21/0x30 [ 8216.116103] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8216.116642] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8216.116840] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8216.117068] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8216.126841] process_one_work+0x2c8/0x700 [ 8216.126864] worker_thread+0x296/0x6e0 [ 8216.126869] ? rescuer_thread+0x570/0x570 [ 8216.126872] kthread+0x1d1/0x200 [ 8216.126877] ? set_kthread_struct+0x70/0x70 [ 8216.126881] ret_from_fork+0x1f/0x30 [ 8216.552966] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8216.557986] ? woken_wake_function+0x30/0x30 [ 8216.559111] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8216.563171] kthread+0x1d1/0x200 [ 8216.564513] ? set_kthread_struct+0x70/0x70 [ 8216.566627] ret_from_fork+0x1f/0x30 [ 8232.056026] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 8232.061516] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8232.062083] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8232.094293] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8232.094566] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8232.094578] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8232.094592] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8232.094596] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8232.094601] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8232.094603] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8232.094605] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8232.094607] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8232.094609] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8232.094613] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8232.094615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8232.094617] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8232.094626] Call Trace: [ 8232.094631] [ 8232.094637] ? show_regs.cold.9+0x22/0x2f [ 8232.094656] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8232.094663] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8232.094667] ? watchdog+0x40/0x40 [ 8232.094671] ? hrtimer_interrupt+0x128/0x2c0 [ 8232.094675] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8232.094683] ? apic_timer_interrupt+0xf/0x20 [ 8232.094687] [ 8232.094690] ? native_safe_halt+0x2e/0x50 [ 8232.094694] kvm_wait+0xcd/0xe0 [ 8232.094702] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8232.094709] do_raw_spin_lock+0x177/0x190 [ 8232.094714] _raw_spin_lock+0x21/0x30 [ 8232.094720] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8232.095625] obd_mod_free+0x46/0x140 [ptlrpc] [ 8232.096162] mdc_commit_open+0x91/0xb0 [mdc] [ 8232.096205] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8232.096550] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8232.097156] after_reply+0xe2c/0x1780 [ptlrpc] [ 8232.097503] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8232.097726] ? __next_timer_interrupt+0x160/0x160 [ 8232.097736] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8232.103659] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8232.103736] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8232.103745] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8232.103748] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8232.103761] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8232.103765] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8232.103770] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8232.103772] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8232.103774] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8232.103776] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8232.103778] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8232.103781] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8232.103783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8232.103785] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8232.103794] Call Trace: [ 8232.103798] [ 8232.103804] ? show_regs.cold.9+0x22/0x2f [ 8232.103814] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8232.103835] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8232.103839] ? watchdog+0x40/0x40 [ 8232.103843] ? hrtimer_interrupt+0x128/0x2c0 [ 8232.103846] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8232.103860] ? apic_timer_interrupt+0xf/0x20 [ 8232.103865] [ 8232.103867] ? native_safe_halt+0x2e/0x50 [ 8232.103871] kvm_wait+0xcd/0xe0 [ 8232.103878] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8232.103885] do_raw_spin_lock+0x177/0x190 [ 8232.103889] _raw_spin_lock+0x21/0x30 [ 8232.103894] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8232.118650] ? do_wait_intr+0xf0/0xf0 [ 8232.118674] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8232.129433] ? do_raw_spin_unlock+0x75/0x190 [ 8232.129456] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8232.134064] kthread+0x1d1/0x200 [ 8232.134082] ? set_kthread_struct+0x70/0x70 [ 8232.134086] ret_from_fork+0x1f/0x30 [ 8232.525665] ? do_wait_intr+0xf0/0xf0 [ 8232.529638] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8232.537663] mdc_reint+0x2d/0x110 [mdc] [ 8232.544423] mdc_unlink+0x658/0x870 [mdc] [ 8232.545905] lmv_unlink+0x34d/0xbd0 [lmv] [ 8232.546990] ll_unlink+0x24d/0x7f0 [lustre] [ 8232.556214] vfs_unlink+0x202/0x330 [ 8232.563492] do_unlinkat+0x2fa/0x460 [ 8232.568703] __x64_sys_unlink+0x37/0x50 [ 8232.575444] do_syscall_64+0xc1/0x440 [ 8232.582210] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8232.583937] RIP: 0033:0x7f83e5f5c10b [ 8232.588197] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8232.603869] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8232.613533] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8232.616533] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8232.625804] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8232.630194] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8232.634491] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8237.544059] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5724s! [ 8237.550197] Showing busy workqueues and worker pools: [ 8237.553249] workqueue events: flags=0x0 [ 8237.558166] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8237.564078] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8237.576305] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8237.584679] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8237.593757] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8237.600475] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8237.605404] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8237.621538] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8237.634628] workqueue events_long: flags=0x0 [ 8237.640401] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8237.659863] pending: lu_objects_flush [obdclass] [ 8237.673208] workqueue events_unbound: flags=0x2 [ 8237.677490] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8237.688484] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8237.695823] workqueue events_power_efficient: flags=0x80 [ 8237.700736] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8237.705955] pending: neigh_periodic_work, do_cache_clean [ 8237.715717] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8237.727155] pending: neigh_periodic_work, check_lifetime [ 8237.741178] workqueue mm_percpu_wq: flags=0x8 [ 8237.750836] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8237.759600] pending: vmstat_update, lru_add_drain_per_cpu [ 8237.767151] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8237.775395] pending: vmstat_update [ 8237.778117] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8237.783506] pending: lru_add_drain_per_cpu BAR(48) [ 8237.790621] workqueue ipv6_addrconf: flags=0x40008 [ 8237.794586] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8237.802720] pending: addrconf_verify_work [ 8237.810726] workqueue knbd0-recv: flags=0x1a [ 8237.818562] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8237.820950] in-flight: 650:recv_work [ 8237.830436] workqueue ptlrpc_pinger: flags=0x2 [ 8237.836516] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8237.844049] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8237.847560] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5724s workers=4 idle: 36 466 29757 [ 8237.855741] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8227s workers=2 idle: 652 [ 8244.059038] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 8244.060028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 8244.066001] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8244.067248] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8244.067255] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8244.067258] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8244.067271] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8244.067275] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8244.067279] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8244.067281] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8244.067283] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8244.067285] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8244.067288] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8244.067291] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8244.067293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8244.067296] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8244.067304] Call Trace: [ 8244.067309] [ 8244.067314] ? show_regs.cold.9+0x22/0x2f [ 8244.067322] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8244.067329] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8244.067333] ? watchdog+0x40/0x40 [ 8244.067337] ? hrtimer_interrupt+0x128/0x2c0 [ 8244.067341] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8244.067347] ? apic_timer_interrupt+0xf/0x20 [ 8244.067352] [ 8244.067354] ? native_safe_halt+0x2e/0x50 [ 8244.067358] kvm_wait+0xcd/0xe0 [ 8244.067365] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8244.067373] do_raw_spin_lock+0x177/0x190 [ 8244.067377] _raw_spin_lock+0x21/0x30 [ 8244.067381] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8244.067639] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8244.075576] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) [ 8244.101737] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8244.102318] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8244.102543] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8244.111715] mgs(O) [ 8244.116932] ? woken_wake_function+0x30/0x30 [ 8244.116964] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8244.117338] kthread+0x1d1/0x200 [ 8244.117345] ? set_kthread_struct+0x70/0x70 [ 8244.117349] ret_from_fork+0x1f/0x30 [ 8244.294568] lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8244.338185] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8244.347574] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8244.352658] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8244.357782] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8244.360419] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8244.377831] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8244.381796] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8244.391510] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8244.398252] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8244.403082] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8244.407468] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8244.410709] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8244.417016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8244.418927] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8244.426466] Call Trace: [ 8244.430784] [ 8244.433776] ? show_regs.cold.9+0x22/0x2f [ 8244.436637] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8244.439840] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8244.442553] ? watchdog+0x40/0x40 [ 8244.444799] ? hrtimer_interrupt+0x128/0x2c0 [ 8244.449512] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8244.455323] ? apic_timer_interrupt+0xf/0x20 [ 8244.456796] [ 8244.459159] ? native_safe_halt+0x2e/0x50 [ 8244.460603] kvm_wait+0xcd/0xe0 [ 8244.461554] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8244.470503] do_raw_spin_lock+0x177/0x190 [ 8244.473274] _raw_spin_lock+0x21/0x30 [ 8244.475039] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8244.479296] process_one_work+0x2c8/0x700 [ 8244.481584] worker_thread+0x296/0x6e0 [ 8244.483117] ? rescuer_thread+0x570/0x570 [ 8244.486479] kthread+0x1d1/0x200 [ 8244.487864] ? set_kthread_struct+0x70/0x70 [ 8244.491445] ret_from_fork+0x1f/0x30 [ 8260.062162] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8260.065850] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8260.086730] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8260.089996] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8260.092702] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8260.095480] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8260.104956] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8260.109630] RAX: 000000000000117c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8260.112577] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8260.115158] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8260.118048] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8260.121366] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8260.123794] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8260.126586] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8260.129595] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8260.131938] Call Trace: [ 8260.133082] [ 8260.134347] ? show_regs.cold.9+0x22/0x2f [ 8260.135735] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8260.137994] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8260.139542] ? watchdog+0x40/0x40 [ 8260.140612] ? hrtimer_interrupt+0x128/0x2c0 [ 8260.141784] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8260.143258] ? apic_timer_interrupt+0xf/0x20 [ 8260.144887] [ 8260.145484] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8260.147100] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8260.148570] do_raw_spin_lock+0x177/0x190 [ 8260.149736] _raw_spin_lock+0x21/0x30 [ 8260.150843] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8260.152997] ? do_raw_spin_unlock+0x75/0x190 [ 8260.154425] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8260.156304] ? do_wait_intr+0xf0/0xf0 [ 8260.157442] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8260.159682] mdc_reint+0x2d/0x110 [mdc] [ 8260.160955] mdc_unlink+0x658/0x870 [mdc] [ 8260.162364] lmv_unlink+0x34d/0xbd0 [lmv] [ 8260.163685] ll_unlink+0x24d/0x7f0 [lustre] [ 8260.165228] vfs_unlink+0x202/0x330 [ 8260.166555] do_unlinkat+0x2fa/0x460 [ 8260.167704] __x64_sys_unlink+0x37/0x50 [ 8260.169161] do_syscall_64+0xc1/0x440 [ 8260.170471] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8260.172247] RIP: 0033:0x7f83e5f5c10b [ 8260.173592] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8260.180143] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8260.182655] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8260.185658] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8260.188581] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8260.191650] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8260.194929] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8264.056300] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 8264.058923] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8264.091500] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8264.098945] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8264.104385] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8264.106877] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8264.116381] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 8264.119203] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8264.122341] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8264.125519] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8264.128983] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8264.132082] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8264.135455] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8264.139308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8264.142289] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8264.145630] Call Trace: [ 8264.146708] [ 8264.147971] ? show_regs.cold.9+0x22/0x2f [ 8264.149770] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8264.152186] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8264.154412] ? watchdog+0x40/0x40 [ 8264.155881] ? hrtimer_interrupt+0x128/0x2c0 [ 8264.158195] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8264.160556] ? apic_timer_interrupt+0xf/0x20 [ 8264.162388] [ 8264.163372] ? native_safe_halt+0x2e/0x50 [ 8264.165465] kvm_wait+0xcd/0xe0 [ 8264.167269] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8264.169952] do_raw_spin_lock+0x177/0x190 [ 8264.172181] _raw_spin_lock+0x21/0x30 [ 8264.173859] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8264.176556] obd_mod_free+0x46/0x140 [ptlrpc] [ 8264.178840] mdc_commit_open+0x91/0xb0 [mdc] [ 8264.180831] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8264.183996] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8264.186877] after_reply+0xe2c/0x1780 [ptlrpc] [ 8264.189878] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8264.192387] ? __next_timer_interrupt+0x160/0x160 [ 8264.194691] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8264.197003] ? do_wait_intr+0xf0/0xf0 [ 8264.198568] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8264.201323] kthread+0x1d1/0x200 [ 8264.202739] ? set_kthread_struct+0x70/0x70 [ 8264.205118] ret_from_fork+0x1f/0x30 [ 8268.259883] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5755s! [ 8268.278879] Showing busy workqueues and worker pools: [ 8268.286143] workqueue events: flags=0x0 [ 8268.293863] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8268.303142] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8268.317489] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8268.330243] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8268.335791] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8268.340867] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8268.352471] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8268.367259] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8268.370455] workqueue events_long: flags=0x0 [ 8268.379906] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8268.387232] pending: lu_objects_flush [obdclass] [ 8268.392409] workqueue events_unbound: flags=0x2 [ 8268.395779] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8268.401399] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8268.411925] workqueue events_power_efficient: flags=0x80 [ 8268.415094] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8268.420203] pending: neigh_periodic_work, do_cache_clean [ 8268.428431] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8268.432690] pending: neigh_periodic_work, check_lifetime [ 8268.435876] workqueue mm_percpu_wq: flags=0x8 [ 8268.445060] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8268.450543] pending: vmstat_update, lru_add_drain_per_cpu [ 8268.455709] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8268.463599] pending: vmstat_update [ 8268.468978] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8268.479437] pending: lru_add_drain_per_cpu BAR(48) [ 8268.481429] workqueue ipv6_addrconf: flags=0x40008 [ 8268.487290] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8268.492554] pending: addrconf_verify_work [ 8268.497335] workqueue knbd0-recv: flags=0x1a [ 8268.503245] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8268.511397] in-flight: 650:recv_work [ 8268.512696] workqueue ptlrpc_pinger: flags=0x2 [ 8268.518684] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8268.521837] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8268.526635] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5755s workers=4 idle: 36 466 29757 [ 8268.536436] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8258s workers=2 idle: 652 [ 8272.059052] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 8272.060223] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 8272.063360] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [ 8272.066897] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [ 8272.078219] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8272.078273] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8272.078280] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8272.078283] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8272.078302] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8272.078307] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8272.078311] RAX: 000000000000138c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8272.078313] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8272.078315] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8272.078317] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8272.078319] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8272.078323] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8272.078326] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8272.078328] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8272.078336] Call Trace: [ 8272.078341] [ 8272.078347] ? show_regs.cold.9+0x22/0x2f [ 8272.078356] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8272.078368] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8272.078373] ? watchdog+0x40/0x40 [ 8272.078378] ? hrtimer_interrupt+0x128/0x2c0 [ 8272.078382] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8272.078390] ? apic_timer_interrupt+0xf/0x20 [ 8272.078395] [ 8272.078398] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8272.078407] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8272.078415] do_raw_spin_lock+0x177/0x190 [ 8272.078419] _raw_spin_lock+0x21/0x30 [ 8272.078423] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8272.078746] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8272.079060] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8272.079548] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8272.079865] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8272.080136] ? woken_wake_function+0x30/0x30 [ 8272.080151] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8272.080398] kthread+0x1d1/0x200 [ 8272.080403] ? set_kthread_struct+0x70/0x70 [ 8272.080506] ret_from_fork+0x1f/0x30 [ 8272.187433] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8272.205462] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8272.210307] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8272.214369] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8272.217302] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8272.220120] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8272.227643] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8272.231475] RAX: 000000000000241e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8272.234408] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8272.237361] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8272.241057] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8272.244355] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8272.248028] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8272.254876] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8272.258059] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8272.261256] Call Trace: [ 8272.262498] [ 8272.263763] ? show_regs.cold.9+0x22/0x2f [ 8272.266220] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8272.268318] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8272.274020] ? watchdog+0x40/0x40 [ 8272.275380] ? hrtimer_interrupt+0x128/0x2c0 [ 8272.277469] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8272.279590] ? apic_timer_interrupt+0xf/0x20 [ 8272.281407] [ 8272.282420] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8272.284653] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8272.287296] do_raw_spin_lock+0x177/0x190 [ 8272.288476] _raw_spin_lock+0x21/0x30 [ 8272.290238] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8272.293081] process_one_work+0x2c8/0x700 [ 8272.294755] worker_thread+0x296/0x6e0 [ 8272.297132] ? rescuer_thread+0x570/0x570 [ 8272.298881] kthread+0x1d1/0x200 [ 8272.301192] ? set_kthread_struct+0x70/0x70 [ 8272.303030] ret_from_fork+0x1f/0x30 [ 8288.065533] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8288.073160] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8288.122475] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8288.146541] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8288.154088] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8288.157871] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8288.186730] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8288.195681] RAX: 0000000000005886 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8288.203647] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8288.210025] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8288.216566] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8288.226070] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8288.237838] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8288.260101] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8288.263388] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8288.266540] Call Trace: [ 8288.267987] [ 8288.268909] ? show_regs.cold.9+0x22/0x2f [ 8288.270590] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8288.272727] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8288.275036] ? watchdog+0x40/0x40 [ 8288.276463] ? hrtimer_interrupt+0x128/0x2c0 [ 8288.278196] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8288.280227] ? apic_timer_interrupt+0xf/0x20 [ 8288.282210] [ 8288.283216] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8288.285448] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8288.288301] do_raw_spin_lock+0x177/0x190 [ 8288.290263] _raw_spin_lock+0x21/0x30 [ 8288.292020] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8288.295108] ? do_raw_spin_unlock+0x75/0x190 [ 8288.296920] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8288.299326] ? do_wait_intr+0xf0/0xf0 [ 8288.300932] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8288.303673] mdc_reint+0x2d/0x110 [mdc] [ 8288.305344] mdc_unlink+0x658/0x870 [mdc] [ 8288.307246] lmv_unlink+0x34d/0xbd0 [lmv] [ 8288.308946] ll_unlink+0x24d/0x7f0 [lustre] [ 8288.311121] vfs_unlink+0x202/0x330 [ 8288.313657] do_unlinkat+0x2fa/0x460 [ 8288.319979] __x64_sys_unlink+0x37/0x50 [ 8288.321649] do_syscall_64+0xc1/0x440 [ 8288.329692] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8288.335475] RIP: 0033:0x7f83e5f5c10b [ 8288.338339] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8288.347569] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8288.351344] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8288.355210] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8288.358454] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8288.361875] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8288.365077] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8296.056141] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 8296.059951] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8296.100100] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8296.120048] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8296.126958] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8296.133515] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8296.159200] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8296.169581] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8296.181465] RDX: 0000000000006e3d RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8296.186597] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8296.197297] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8296.208903] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8296.220093] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8296.231580] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8296.234888] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8296.241221] Call Trace: [ 8296.242397] [ 8296.243489] ? show_regs.cold.9+0x22/0x2f [ 8296.246210] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8296.258880] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8296.260538] ? watchdog+0x40/0x40 [ 8296.270114] ? hrtimer_interrupt+0x128/0x2c0 [ 8296.271737] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8296.273439] ? apic_timer_interrupt+0xf/0x20 [ 8296.285927] [ 8296.286716] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8296.302579] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8296.309000] do_raw_spin_lock+0x177/0x190 [ 8296.312337] _raw_spin_lock+0x21/0x30 [ 8296.318464] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8296.324947] obd_mod_free+0x46/0x140 [ptlrpc] [ 8296.328468] mdc_commit_open+0x91/0xb0 [mdc] [ 8296.331395] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8296.333412] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8296.342024] after_reply+0xe2c/0x1780 [ptlrpc] [ 8296.346115] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8296.349048] ? __next_timer_interrupt+0x160/0x160 [ 8296.351664] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8296.357384] ? do_wait_intr+0xf0/0xf0 [ 8296.360621] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8296.372035] kthread+0x1d1/0x200 [ 8296.373182] ? set_kthread_struct+0x70/0x70 [ 8296.376536] ret_from_fork+0x1f/0x30 [ 8298.981908] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5785s! [ 8299.000432] Showing busy workqueues and worker pools: [ 8299.002340] workqueue events: flags=0x0 [ 8299.006118] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8299.010270] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8299.033770] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8299.038680] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8299.044147] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8299.054148] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8299.060137] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8299.067603] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8299.075431] workqueue events_long: flags=0x0 [ 8299.085677] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8299.087936] pending: lu_objects_flush [obdclass] [ 8299.100343] workqueue events_unbound: flags=0x2 [ 8299.107424] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8299.113458] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8299.117109] workqueue events_power_efficient: flags=0x80 [ 8299.119831] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8299.125647] pending: neigh_periodic_work, do_cache_clean [ 8299.130845] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8299.141139] pending: neigh_periodic_work, check_lifetime [ 8299.149007] workqueue mm_percpu_wq: flags=0x8 [ 8299.153086] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8299.157279] pending: vmstat_update, lru_add_drain_per_cpu [ 8299.171644] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8299.185067] pending: vmstat_update [ 8299.189835] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8299.209737] pending: lru_add_drain_per_cpu BAR(48) [ 8299.215813] workqueue ipv6_addrconf: flags=0x40008 [ 8299.219909] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8299.224085] pending: addrconf_verify_work [ 8299.233987] workqueue knbd0-recv: flags=0x1a [ 8299.238011] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8299.245092] in-flight: 650:recv_work [ 8299.249542] workqueue ptlrpc_pinger: flags=0x2 [ 8299.254409] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8299.258934] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8299.265634] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5786s workers=4 idle: 36 466 29757 [ 8299.272769] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8289s workers=2 idle: 652 [ 8300.060090] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 8300.062117] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 8300.064969] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8300.065047] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8300.065053] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8300.065059] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8300.065738] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8300.065893] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8300.065899] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8300.065904] RAX: 0000000000005414 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8300.065968] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8300.065971] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8300.065973] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8300.065975] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8300.065978] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8300.065981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8300.065983] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8300.065990] Call Trace: [ 8300.066001] [ 8300.066007] ? show_regs.cold.9+0x22/0x2f [ 8300.066019] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8300.066026] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8300.066030] ? watchdog+0x40/0x40 [ 8300.066034] ? hrtimer_interrupt+0x128/0x2c0 [ 8300.066037] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8300.066045] ? apic_timer_interrupt+0xf/0x20 [ 8300.066050] [ 8300.066052] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8300.066056] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8300.066062] do_raw_spin_lock+0x177/0x190 [ 8300.066066] _raw_spin_lock+0x21/0x30 [ 8300.066130] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8300.066571] process_one_work+0x2c8/0x700 [ 8300.066580] worker_thread+0x296/0x6e0 [ 8300.066585] ? rescuer_thread+0x570/0x570 [ 8300.066589] kthread+0x1d1/0x200 [ 8300.066594] ? set_kthread_struct+0x70/0x70 [ 8300.066819] ret_from_fork+0x1f/0x30 [ 8300.446000] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8300.495704] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8300.506573] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8300.511260] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8300.523519] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8300.544905] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8300.551936] RAX: 0000000000007215 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8300.555606] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8300.563419] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8300.570412] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8300.574362] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8300.576985] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8300.583994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8300.587479] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8300.592368] Call Trace: [ 8300.594109] [ 8300.594951] ? show_regs.cold.9+0x22/0x2f [ 8300.597116] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8300.599503] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8300.601693] ? watchdog+0x40/0x40 [ 8300.603701] ? hrtimer_interrupt+0x128/0x2c0 [ 8300.607089] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8300.609723] ? apic_timer_interrupt+0xf/0x20 [ 8300.611890] [ 8300.612848] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8300.615786] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8300.619257] do_raw_spin_lock+0x177/0x190 [ 8300.621407] _raw_spin_lock+0x21/0x30 [ 8300.623291] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8300.627268] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8300.630723] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8300.635748] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8300.640580] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8300.647881] ? woken_wake_function+0x30/0x30 [ 8300.650608] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8300.654279] kthread+0x1d1/0x200 [ 8300.658598] ? set_kthread_struct+0x70/0x70 [ 8300.661013] ret_from_fork+0x1f/0x30 [ 8316.065625] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8316.071453] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8316.088847] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8316.093201] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8316.096173] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8316.098033] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8316.105698] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8316.110230] RAX: 0000000000006758 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8316.114413] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8316.118071] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8316.128081] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8316.130780] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8316.134053] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8316.141725] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8316.149928] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8316.153937] Call Trace: [ 8316.155428] [ 8316.156545] ? show_regs.cold.9+0x22/0x2f [ 8316.158744] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8316.161797] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8316.164548] ? watchdog+0x40/0x40 [ 8316.166416] ? hrtimer_interrupt+0x128/0x2c0 [ 8316.168697] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8316.172116] ? apic_timer_interrupt+0xf/0x20 [ 8316.173722] [ 8316.174510] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8316.177838] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8316.180894] do_raw_spin_lock+0x177/0x190 [ 8316.187853] _raw_spin_lock+0x21/0x30 [ 8316.192740] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8316.194600] ? do_raw_spin_unlock+0x75/0x190 [ 8316.196150] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8316.198200] ? do_wait_intr+0xf0/0xf0 [ 8316.201136] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8316.205195] mdc_reint+0x2d/0x110 [mdc] [ 8316.208974] mdc_unlink+0x658/0x870 [mdc] [ 8316.211901] lmv_unlink+0x34d/0xbd0 [lmv] [ 8316.214278] ll_unlink+0x24d/0x7f0 [lustre] [ 8316.217966] vfs_unlink+0x202/0x330 [ 8316.219898] do_unlinkat+0x2fa/0x460 [ 8316.222249] __x64_sys_unlink+0x37/0x50 [ 8316.224297] do_syscall_64+0xc1/0x440 [ 8316.227351] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8316.231819] RIP: 0033:0x7f83e5f5c10b [ 8316.234958] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8316.249132] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8316.254020] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8316.257991] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8316.271075] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8316.280221] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8316.292220] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8324.056063] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 8324.061507] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8324.125777] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8324.141140] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8324.159591] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8324.164975] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8324.189860] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8324.201749] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8324.215581] RDX: 0000000000004f0b RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8324.224546] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8324.235786] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8324.247041] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8324.259979] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8324.268879] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8324.277031] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8324.280946] Call Trace: [ 8324.287035] [ 8324.289119] ? show_regs.cold.9+0x22/0x2f [ 8324.297990] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8324.303803] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8324.313856] ? watchdog+0x40/0x40 [ 8324.320903] ? hrtimer_interrupt+0x128/0x2c0 [ 8324.328651] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8324.338121] ? apic_timer_interrupt+0xf/0x20 [ 8324.340505] [ 8324.346391] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8324.353076] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8324.358608] do_raw_spin_lock+0x177/0x190 [ 8324.363621] _raw_spin_lock+0x21/0x30 [ 8324.367744] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8324.373176] obd_mod_free+0x46/0x140 [ptlrpc] [ 8324.380377] mdc_commit_open+0x91/0xb0 [mdc] [ 8324.383249] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8324.393445] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8324.402518] after_reply+0xe2c/0x1780 [ptlrpc] [ 8324.408172] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8324.423592] ? __next_timer_interrupt+0x160/0x160 [ 8324.425310] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8324.432191] ? do_wait_intr+0xf0/0xf0 [ 8324.434177] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8324.439781] kthread+0x1d1/0x200 [ 8324.445417] ? set_kthread_struct+0x70/0x70 [ 8324.446987] ret_from_fork+0x1f/0x30 [ 8328.063194] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 8328.067027] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 8328.080294] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8328.080539] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8328.080547] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8328.080550] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8328.080564] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8328.080569] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8328.080574] RAX: 0000000000007d83 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8328.080576] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8328.080578] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8328.080580] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8328.080583] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8328.080586] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8328.080588] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8328.080590] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8328.080599] Call Trace: [ 8328.080604] [ 8328.080610] ? show_regs.cold.9+0x22/0x2f [ 8328.080622] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8328.080629] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8328.080633] ? watchdog+0x40/0x40 [ 8328.080637] ? hrtimer_interrupt+0x128/0x2c0 [ 8328.080641] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8328.080650] ? apic_timer_interrupt+0xf/0x20 [ 8328.080655] [ 8328.080658] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8328.080663] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8328.080669] do_raw_spin_lock+0x177/0x190 [ 8328.080674] _raw_spin_lock+0x21/0x30 [ 8328.080678] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8328.080995] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8328.081568] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8328.081787] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8328.082993] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8328.083218] ? woken_wake_function+0x30/0x30 [ 8328.083228] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8328.086392] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8328.086466] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8328.086471] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8328.086477] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8328.086749] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8328.086760] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8328.086763] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8328.086767] RAX: 0000000000007144 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8328.086769] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8328.086771] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8328.086773] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8328.086776] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8328.086779] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8328.086782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8328.086784] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8328.086793] Call Trace: [ 8328.086799] [ 8328.086803] ? show_regs.cold.9+0x22/0x2f [ 8328.086813] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8328.086819] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8328.086824] ? watchdog+0x40/0x40 [ 8328.086827] ? hrtimer_interrupt+0x128/0x2c0 [ 8328.086831] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8328.086839] ? apic_timer_interrupt+0xf/0x20 [ 8328.086843] [ 8328.086847] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8328.086852] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8328.086858] do_raw_spin_lock+0x177/0x190 [ 8328.086862] _raw_spin_lock+0x21/0x30 [ 8328.086866] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8328.087014] process_one_work+0x2c8/0x700 [ 8328.087019] worker_thread+0x296/0x6e0 [ 8328.087022] ? rescuer_thread+0x570/0x570 [ 8328.087025] kthread+0x1d1/0x200 [ 8328.087028] ? set_kthread_struct+0x70/0x70 [ 8328.087030] ret_from_fork+0x1f/0x30 [ 8328.846435] kthread+0x1d1/0x200 [ 8328.848659] ? set_kthread_struct+0x70/0x70 [ 8328.851650] ret_from_fork+0x1f/0x30 [ 8329.697046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5816s! [ 8329.708992] Showing busy workqueues and worker pools: [ 8329.711714] workqueue events: flags=0x0 [ 8329.713053] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8329.721106] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8329.725701] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8329.728463] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8329.735446] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8329.742037] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8329.745924] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8329.754909] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8329.758638] workqueue events_long: flags=0x0 [ 8329.759962] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8329.769199] pending: lu_objects_flush [obdclass] [ 8329.775020] workqueue events_unbound: flags=0x2 [ 8329.781664] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8329.784659] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8329.792192] workqueue events_power_efficient: flags=0x80 [ 8329.796011] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8329.802702] pending: neigh_periodic_work, do_cache_clean [ 8329.805503] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8329.808503] pending: neigh_periodic_work, check_lifetime [ 8329.813709] workqueue mm_percpu_wq: flags=0x8 [ 8329.821195] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8329.826475] pending: vmstat_update, lru_add_drain_per_cpu [ 8329.830142] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8329.839743] pending: vmstat_update [ 8329.840904] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8329.844904] pending: lru_add_drain_per_cpu BAR(48) [ 8329.846174] workqueue ipv6_addrconf: flags=0x40008 [ 8329.848977] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8329.854124] pending: addrconf_verify_work [ 8329.857529] workqueue knbd0-recv: flags=0x1a [ 8329.858816] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8329.863890] in-flight: 650:recv_work [ 8329.868968] workqueue ptlrpc_pinger: flags=0x2 [ 8329.873123] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8329.881916] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8329.883917] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5816s workers=4 idle: 36 466 29757 [ 8329.901196] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8319s workers=2 idle: 652 [ 8341.112222] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 8341.118283] rcu: 0-....: (4046930 ticks this GP) idle=30a/1/0x4000000000000000 softirq=128519/128519 fqs=126395 [ 8341.127547] (detected by 1, t=6360153 jiffies, g=262441, q=59043) [ 8341.136695] Sending NMI from CPU 1 to CPUs 0: [ 8341.140395] NMI backtrace for cpu 0 [ 8341.140403] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8341.140405] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8341.140407] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8341.140409] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8341.140412] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 [ 8341.140415] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8341.140416] RDX: 0000000000006c3b RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8341.140418] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8341.140419] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8341.140420] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8341.140422] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8341.140423] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8341.140424] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8341.140426] Call Trace: [ 8341.140427] [ 8341.140428] ? show_regs.cold.9+0x22/0x2f [ 8341.140429] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8341.140430] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8341.140431] ? nmi_handle+0xbd/0x210 [ 8341.140432] ? default_do_nmi+0x4b/0x1b0 [ 8341.140434] ? do_nmi+0x280/0x380 [ 8341.140435] ? end_repeat_nmi+0x16/0x1f [ 8341.140436] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8341.140437] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8341.140438] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8341.140439] [ 8341.140440] do_raw_spin_lock+0x177/0x190 [ 8341.140441] _raw_spin_lock+0x21/0x30 [ 8341.140442] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8341.140443] obd_mod_free+0x46/0x140 [ptlrpc] [ 8341.140445] mdc_commit_open+0x91/0xb0 [mdc] [ 8341.140446] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8341.140448] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8341.140449] after_reply+0xe2c/0x1780 [ptlrpc] [ 8341.140450] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8341.140451] ? __next_timer_interrupt+0x160/0x160 [ 8341.140452] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8341.140453] ? do_wait_intr+0xf0/0xf0 [ 8341.140454] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8341.140455] kthread+0x1d1/0x200 [ 8341.140456] ? set_kthread_struct+0x70/0x70 [ 8341.140457] ret_from_fork+0x1f/0x30 [ 8341.141576] rcu: rcu_sched kthread starved for 5829340 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 8341.397639] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8341.404792] rcu: RCU grace-period kthread stack dump: [ 8341.406515] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8341.410917] Call Trace: [ 8341.413666] __schedule+0x351/0xcb0 [ 8341.417470] schedule+0xc0/0x180 [ 8341.421398] schedule_timeout+0xb4/0x190 [ 8341.424784] ? do_raw_spin_unlock+0x75/0x190 [ 8341.427394] ? __next_timer_interrupt+0x160/0x160 [ 8341.429774] ? prepare_to_swait_event+0xf3/0x140 [ 8341.432443] rcu_gp_kthread+0x8dd/0x15b0 [ 8341.434460] ? force_qs_rnp+0x2c0/0x2c0 [ 8341.436712] kthread+0x1d1/0x200 [ 8341.438387] ? set_kthread_struct+0x70/0x70 [ 8341.440554] ret_from_fork+0x1f/0x30 [ 8341.442359] rcu: Stack dump where RCU GP kthread last ran: [ 8341.445353] Sending NMI from CPU 1 to CPUs 3: [ 8341.448784] NMI backtrace for cpu 3 [ 8341.448790] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8341.448792] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8341.448794] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8341.448795] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8341.448798] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 [ 8341.448801] RAX: 0000000000004327 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8341.448802] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8341.448803] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8341.448805] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8341.448806] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8341.448808] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8341.448809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8341.448810] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8341.448812] Call Trace: [ 8341.448813] [ 8341.448814] ? show_regs.cold.9+0x22/0x2f [ 8341.448815] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8341.448816] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8341.448817] ? nmi_handle+0xbd/0x210 [ 8341.448818] ? default_do_nmi+0x4b/0x1b0 [ 8341.448818] ? do_nmi+0x280/0x380 [ 8341.448820] ? end_repeat_nmi+0x16/0x1f [ 8341.448821] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8341.448822] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8341.448823] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8341.448828] [ 8341.448829] do_raw_spin_lock+0x177/0x190 [ 8341.448830] _raw_spin_lock+0x21/0x30 [ 8341.448831] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8341.448833] ? do_raw_spin_unlock+0x75/0x190 [ 8341.448834] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8341.448835] ? do_wait_intr+0xf0/0xf0 [ 8341.448836] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8341.448837] mdc_reint+0x2d/0x110 [mdc] [ 8341.448838] mdc_unlink+0x658/0x870 [mdc] [ 8341.448839] lmv_unlink+0x34d/0xbd0 [lmv] [ 8341.448840] ll_unlink+0x24d/0x7f0 [lustre] [ 8341.448841] vfs_unlink+0x202/0x330 [ 8341.448842] do_unlinkat+0x2fa/0x460 [ 8341.448843] __x64_sys_unlink+0x37/0x50 [ 8341.448844] do_syscall_64+0xc1/0x440 [ 8341.448845] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8341.448846] RIP: 0033:0x7f83e5f5c10b [ 8341.448847] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8341.448850] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8341.448852] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8341.448853] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8341.448854] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8341.448856] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8341.448857] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8356.060709] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 8356.069679] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8356.114343] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8356.132305] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8356.137075] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8356.143568] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8356.146610] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8356.157582] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8356.160866] RAX: 0000000000005309 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8356.163702] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8356.166737] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8356.169742] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8356.172474] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8356.175932] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8356.179519] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8356.182039] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8356.185382] Call Trace: [ 8356.186503] [ 8356.187482] ? show_regs.cold.9+0x22/0x2f [ 8356.189190] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8356.191088] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8356.192843] ? watchdog+0x40/0x40 [ 8356.194238] ? hrtimer_interrupt+0x128/0x2c0 [ 8356.196421] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8356.198414] ? apic_timer_interrupt+0xf/0x20 [ 8356.200498] [ 8356.201443] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8356.204725] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8356.207530] do_raw_spin_lock+0x177/0x190 [ 8356.209458] _raw_spin_lock+0x21/0x30 [ 8356.211374] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8356.213772] process_one_work+0x2c8/0x700 [ 8356.215617] worker_thread+0x296/0x6e0 [ 8356.217420] ? rescuer_thread+0x570/0x570 [ 8356.219062] kthread+0x1d1/0x200 [ 8356.220789] ? set_kthread_struct+0x70/0x70 [ 8356.222542] ret_from_fork+0x1f/0x30 [ 8360.416223] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5847s! [ 8360.432009] Showing busy workqueues and worker pools: [ 8360.438878] workqueue events: flags=0x0 [ 8360.446863] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8360.457410] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8360.467947] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8360.477529] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8360.482624] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8360.488358] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8360.490569] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8360.509546] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8360.519528] workqueue events_long: flags=0x0 [ 8360.522767] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8360.527809] pending: lu_objects_flush [obdclass] [ 8360.532483] workqueue events_unbound: flags=0x2 [ 8360.534199] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8360.540385] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8360.544457] workqueue events_power_efficient: flags=0x80 [ 8360.550913] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8360.560270] pending: neigh_periodic_work, do_cache_clean [ 8360.569140] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8360.576489] pending: neigh_periodic_work, check_lifetime [ 8360.585245] workqueue mm_percpu_wq: flags=0x8 [ 8360.588264] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8360.607049] pending: vmstat_update, lru_add_drain_per_cpu [ 8360.616257] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8360.627963] pending: vmstat_update [ 8360.636934] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8360.645980] pending: lru_add_drain_per_cpu BAR(48) [ 8360.655922] workqueue ipv6_addrconf: flags=0x40008 [ 8360.661544] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8360.667018] pending: addrconf_verify_work [ 8360.677347] workqueue knbd0-recv: flags=0x1a [ 8360.678835] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8360.687765] in-flight: 650:recv_work [ 8360.693765] workqueue ptlrpc_pinger: flags=0x2 [ 8360.702310] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8360.718346] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8360.729346] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5847s workers=4 idle: 36 466 29757 [ 8360.741715] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8350s workers=2 idle: 652 [ 8368.059017] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 8368.062757] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8368.066513] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8368.066597] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8368.066607] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8368.066610] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8368.066626] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8368.066630] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8368.066635] RAX: 0000000000004dde RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8368.066638] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8368.066640] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8368.066642] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8368.066644] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8368.066647] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8368.066651] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8368.066653] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8368.066662] Call Trace: [ 8368.066727] [ 8368.071668] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8368.071742] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8368.071750] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8368.071753] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8368.071767] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8368.071770] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8368.071774] RAX: 0000000000002adb RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8368.071776] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8368.071789] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8368.071791] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8368.071794] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8368.071796] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8368.071799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8368.071801] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8368.071809] Call Trace: [ 8368.071815] [ 8368.071821] ? show_regs.cold.9+0x22/0x2f [ 8368.071831] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8368.071837] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8368.071841] ? watchdog+0x40/0x40 [ 8368.071845] ? hrtimer_interrupt+0x128/0x2c0 [ 8368.071849] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8368.071857] ? apic_timer_interrupt+0xf/0x20 [ 8368.071862] [ 8368.071864] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8368.071870] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8368.071874] do_raw_spin_lock+0x177/0x190 [ 8368.071879] _raw_spin_lock+0x21/0x30 [ 8368.071885] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8368.072160] ? do_raw_spin_unlock+0x75/0x190 [ 8368.072166] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8368.072360] ? do_wait_intr+0xf0/0xf0 [ 8368.072367] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8368.072563] mdc_reint+0x2d/0x110 [mdc] [ 8368.072599] mdc_unlink+0x658/0x870 [mdc] [ 8368.072629] lmv_unlink+0x34d/0xbd0 [lmv] [ 8368.072658] ll_unlink+0x24d/0x7f0 [lustre] [ 8368.072761] vfs_unlink+0x202/0x330 [ 8368.072768] do_unlinkat+0x2fa/0x460 [ 8368.072773] __x64_sys_unlink+0x37/0x50 [ 8368.072776] do_syscall_64+0xc1/0x440 [ 8368.072798] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8368.072803] RIP: 0033:0x7f83e5f5c10b [ 8368.072809] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8368.072813] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8368.072817] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8368.072819] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8368.072821] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8368.072824] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8368.072826] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8368.412493] ? show_regs.cold.9+0x22/0x2f [ 8368.416278] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8368.418316] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8368.424178] ? watchdog+0x40/0x40 [ 8368.426524] ? hrtimer_interrupt+0x128/0x2c0 [ 8368.435285] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8368.440454] ? apic_timer_interrupt+0xf/0x20 [ 8368.444904] [ 8368.445917] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8368.450278] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8368.463600] do_raw_spin_lock+0x177/0x190 [ 8368.470175] _raw_spin_lock+0x21/0x30 [ 8368.471577] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8368.481396] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8368.490614] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8368.498561] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8368.502204] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8368.505896] ? woken_wake_function+0x30/0x30 [ 8368.507927] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8368.511184] kthread+0x1d1/0x200 [ 8368.512793] ? set_kthread_struct+0x70/0x70 [ 8368.514524] ret_from_fork+0x1f/0x30 [ 8384.060885] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 8384.063830] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8384.096370] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8384.101375] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8384.105916] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8384.109486] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8384.113066] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8384.123454] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8384.126421] RAX: 0000000000007c98 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8384.130122] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8384.135140] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8384.139497] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8384.144811] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8384.149384] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8384.152479] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8384.154602] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8384.159264] Call Trace: [ 8384.160261] [ 8384.161374] ? show_regs.cold.9+0x22/0x2f [ 8384.164238] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8384.166373] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8384.170410] ? watchdog+0x40/0x40 [ 8384.173841] ? hrtimer_interrupt+0x128/0x2c0 [ 8384.175325] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8384.178263] ? apic_timer_interrupt+0xf/0x20 [ 8384.182231] [ 8384.184442] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8384.190230] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8384.195281] do_raw_spin_lock+0x177/0x190 [ 8384.199048] _raw_spin_lock+0x21/0x30 [ 8384.201142] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8384.204840] process_one_work+0x2c8/0x700 [ 8384.207855] worker_thread+0x296/0x6e0 [ 8384.210226] ? rescuer_thread+0x570/0x570 [ 8384.213019] kthread+0x1d1/0x200 [ 8384.215811] ? set_kthread_struct+0x70/0x70 [ 8384.217476] ret_from_fork+0x1f/0x30 [ 8388.058282] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 8388.068329] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8388.134222] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8388.148823] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8388.154084] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8388.162094] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8388.185678] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8388.195295] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8388.202242] RDX: 0000000000003fc9 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8388.206101] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8388.214992] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8388.219610] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8388.225157] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8388.232433] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8388.238261] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8388.245574] Call Trace: [ 8388.247179] [ 8388.251771] ? show_regs.cold.9+0x22/0x2f [ 8388.254468] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8388.257471] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8388.264326] ? watchdog+0x40/0x40 [ 8388.270075] ? hrtimer_interrupt+0x128/0x2c0 [ 8388.272986] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8388.275659] ? apic_timer_interrupt+0xf/0x20 [ 8388.277819] [ 8388.278523] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8388.284435] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8388.290666] do_raw_spin_lock+0x177/0x190 [ 8388.294004] _raw_spin_lock+0x21/0x30 [ 8388.296023] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8388.298039] obd_mod_free+0x46/0x140 [ptlrpc] [ 8388.303387] mdc_commit_open+0x91/0xb0 [mdc] [ 8388.307917] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8388.310112] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8388.324438] after_reply+0xe2c/0x1780 [ptlrpc] [ 8388.335210] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8388.336882] ? __next_timer_interrupt+0x160/0x160 [ 8388.346436] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8388.353377] ? do_wait_intr+0xf0/0xf0 [ 8388.356627] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8388.360050] kthread+0x1d1/0x200 [ 8388.361927] ? set_kthread_struct+0x70/0x70 [ 8388.367069] ret_from_fork+0x1f/0x30 [ 8391.141503] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5877s! [ 8391.150421] Showing busy workqueues and worker pools: [ 8391.155014] workqueue events: flags=0x0 [ 8391.157276] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8391.164311] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8391.168808] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8391.172252] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8391.175456] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8391.179472] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8391.182938] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8391.185940] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8391.191170] workqueue events_long: flags=0x0 [ 8391.198144] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8391.200488] pending: lu_objects_flush [obdclass] [ 8391.211665] workqueue events_unbound: flags=0x2 [ 8391.213532] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8391.219073] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8391.224899] workqueue events_power_efficient: flags=0x80 [ 8391.229021] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8391.232925] pending: neigh_periodic_work, do_cache_clean [ 8391.239416] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8391.242430] pending: neigh_periodic_work, check_lifetime [ 8391.244924] workqueue mm_percpu_wq: flags=0x8 [ 8391.247104] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8391.256716] pending: vmstat_update, lru_add_drain_per_cpu [ 8391.261226] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8391.264884] pending: vmstat_update [ 8391.266970] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8391.273998] pending: lru_add_drain_per_cpu BAR(48) [ 8391.276972] workqueue ipv6_addrconf: flags=0x40008 [ 8391.278456] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8391.281853] pending: addrconf_verify_work [ 8391.284212] workqueue knbd0-recv: flags=0x1a [ 8391.286222] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8391.289477] in-flight: 650:recv_work [ 8391.291665] workqueue ptlrpc_pinger: flags=0x2 [ 8391.293527] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8391.298038] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8391.303521] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5878s workers=4 idle: 36 466 29757 [ 8391.311151] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8381s workers=2 idle: 652 [ 8396.059056] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 8396.062042] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8396.064038] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) [ 8396.068052] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8396.068137] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8396.068144] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8396.068147] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8396.068162] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8396.068166] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8396.068171] RAX: 0000000000000b12 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8396.068173] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8396.068175] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8396.068177] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8396.068180] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8396.068183] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8396.068186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8396.068188] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8396.068466] Call Trace: [ 8396.068473] [ 8396.068479] ? show_regs.cold.9+0x22/0x2f [ 8396.068489] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8396.068495] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8396.068499] ? watchdog+0x40/0x40 [ 8396.068519] ? hrtimer_interrupt+0x128/0x2c0 [ 8396.068524] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8396.068532] ? apic_timer_interrupt+0xf/0x20 [ 8396.068537] [ 8396.068540] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8396.068545] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8396.068554] do_raw_spin_lock+0x177/0x190 [ 8396.068562] _raw_spin_lock+0x21/0x30 [ 8396.068569] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8396.068877] ? do_raw_spin_unlock+0x75/0x190 [ 8396.068883] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8396.069434] ? do_wait_intr+0xf0/0xf0 [ 8396.069445] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8396.069764] mdc_reint+0x2d/0x110 [mdc] [ 8396.069805] mdc_unlink+0x658/0x870 [mdc] [ 8396.069829] lmv_unlink+0x34d/0xbd0 [lmv] [ 8396.069859] ll_unlink+0x24d/0x7f0 [lustre] [ 8396.070000] vfs_unlink+0x202/0x330 [ 8396.070008] do_unlinkat+0x2fa/0x460 [ 8396.070017] __x64_sys_unlink+0x37/0x50 [ 8396.070021] do_syscall_64+0xc1/0x440 [ 8396.070027] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8396.070033] RIP: 0033:0x7f83e5f5c10b [ 8396.070451] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8396.070458] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8396.070468] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8396.070470] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8396.070472] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8396.070474] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8396.070476] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8396.273979] ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8396.290964] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8396.296840] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8396.301341] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8396.304512] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8396.314900] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8396.318903] RAX: 0000000000007491 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8396.322278] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8396.326341] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8396.330046] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8396.332919] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8396.336485] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8396.340758] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8396.343499] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8396.346698] Call Trace: [ 8396.348023] [ 8396.349213] ? show_regs.cold.9+0x22/0x2f [ 8396.351161] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8396.353365] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8396.355929] ? watchdog+0x40/0x40 [ 8396.357583] ? hrtimer_interrupt+0x128/0x2c0 [ 8396.359655] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8396.361922] ? apic_timer_interrupt+0xf/0x20 [ 8396.364526] [ 8396.365908] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8396.368604] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8396.371321] do_raw_spin_lock+0x177/0x190 [ 8396.373990] _raw_spin_lock+0x21/0x30 [ 8396.376004] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8396.378906] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8396.381894] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8396.387500] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8396.391654] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8396.394170] ? woken_wake_function+0x30/0x30 [ 8396.398972] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8396.401952] kthread+0x1d1/0x200 [ 8396.403474] ? set_kthread_struct+0x70/0x70 [ 8396.405762] ret_from_fork+0x1f/0x30 [ 8412.070204] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 8412.084270] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8412.199036] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8412.203312] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8412.206783] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8412.209422] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8412.211479] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8412.217777] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8412.220056] RAX: 0000000000006a10 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8412.222142] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8412.224612] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8412.227188] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8412.229396] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8412.231245] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8412.233311] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8412.235204] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8412.236878] Call Trace: [ 8412.237407] [ 8412.237927] ? show_regs.cold.9+0x22/0x2f [ 8412.239152] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8412.242936] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8412.245546] ? watchdog+0x40/0x40 [ 8412.246555] ? hrtimer_interrupt+0x128/0x2c0 [ 8412.248989] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8412.251920] ? apic_timer_interrupt+0xf/0x20 [ 8412.254920] [ 8412.255978] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8412.259053] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8412.262076] do_raw_spin_lock+0x177/0x190 [ 8412.269330] _raw_spin_lock+0x21/0x30 [ 8412.273737] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8412.279011] process_one_work+0x2c8/0x700 [ 8412.281894] worker_thread+0x296/0x6e0 [ 8412.284689] ? rescuer_thread+0x570/0x570 [ 8412.287679] kthread+0x1d1/0x200 [ 8412.289814] ? set_kthread_struct+0x70/0x70 [ 8412.291522] ret_from_fork+0x1f/0x30 [ 8416.056024] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 8416.059262] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8416.101985] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8416.109889] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8416.114929] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8416.118157] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8416.136662] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8416.140174] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8416.146464] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8416.154591] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8416.159415] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8416.163138] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8416.168723] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8416.173424] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8416.176410] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8416.179725] Call Trace: [ 8416.181872] [ 8416.183714] ? show_regs.cold.9+0x22/0x2f [ 8416.186783] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8416.192757] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8416.195162] ? watchdog+0x40/0x40 [ 8416.200345] ? hrtimer_interrupt+0x128/0x2c0 [ 8416.201901] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8416.208537] ? apic_timer_interrupt+0xf/0x20 [ 8416.210062] [ 8416.223322] ? native_safe_halt+0x2e/0x50 [ 8416.224789] kvm_wait+0xcd/0xe0 [ 8416.225973] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8416.244852] do_raw_spin_lock+0x177/0x190 [ 8416.246036] _raw_spin_lock+0x21/0x30 [ 8416.255325] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8416.259903] obd_mod_free+0x46/0x140 [ptlrpc] [ 8416.265227] mdc_commit_open+0x91/0xb0 [mdc] [ 8416.266800] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8416.283119] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8416.288933] after_reply+0xe2c/0x1780 [ptlrpc] [ 8416.294839] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8416.300349] ? __next_timer_interrupt+0x160/0x160 [ 8416.303804] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8416.306464] ? do_wait_intr+0xf0/0xf0 [ 8416.308339] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8416.312217] kthread+0x1d1/0x200 [ 8416.316335] ? set_kthread_struct+0x70/0x70 [ 8416.319137] ret_from_fork+0x1f/0x30 [ 8421.345043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5908s! [ 8421.348518] Showing busy workqueues and worker pools: [ 8421.351170] workqueue events: flags=0x0 [ 8421.352978] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8421.356124] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8421.360178] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8421.363740] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8421.366862] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8421.370022] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8421.373813] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8421.376701] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8421.380710] workqueue events_long: flags=0x0 [ 8421.382777] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8421.386123] pending: lu_objects_flush [obdclass] [ 8421.389476] workqueue events_unbound: flags=0x2 [ 8421.392059] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8421.395047] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8421.398448] workqueue events_power_efficient: flags=0x80 [ 8421.402112] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8421.406049] pending: neigh_periodic_work, do_cache_clean [ 8421.408924] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8421.412626] pending: neigh_periodic_work, check_lifetime [ 8421.415850] workqueue mm_percpu_wq: flags=0x8 [ 8421.421071] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8421.426124] pending: vmstat_update, lru_add_drain_per_cpu [ 8421.430129] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8421.438105] pending: vmstat_update [ 8421.442883] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8421.451941] pending: lru_add_drain_per_cpu BAR(48) [ 8421.463717] workqueue ipv6_addrconf: flags=0x40008 [ 8421.472316] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8421.486813] pending: addrconf_verify_work [ 8421.494538] workqueue knbd0-recv: flags=0x1a [ 8421.501477] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8421.506132] in-flight: 650:recv_work [ 8421.512240] workqueue ptlrpc_pinger: flags=0x2 [ 8421.516217] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8421.518643] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8421.521943] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5908s workers=4 idle: 36 466 29757 [ 8421.528094] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8411s workers=2 idle: 652 [ 8424.061131] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 8424.062057] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8424.067922] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) [ 8424.072232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8424.072320] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8424.072328] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8424.072330] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8424.072369] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8424.072374] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8424.072379] RAX: 0000000000004229 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8424.072381] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8424.072384] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8424.072386] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8424.072388] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8424.072396] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8424.072400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8424.072403] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8424.072410] Call Trace: [ 8424.072428] [ 8424.072434] ? show_regs.cold.9+0x22/0x2f [ 8424.072590] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8424.072598] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8424.072602] ? watchdog+0x40/0x40 [ 8424.072606] ? hrtimer_interrupt+0x128/0x2c0 [ 8424.072610] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8424.072618] ? apic_timer_interrupt+0xf/0x20 [ 8424.072623] [ 8424.072626] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8424.072630] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8424.072635] do_raw_spin_lock+0x177/0x190 [ 8424.072639] _raw_spin_lock+0x21/0x30 [ 8424.072645] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8424.072940] ? do_raw_spin_unlock+0x75/0x190 [ 8424.072946] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8424.077930] ? do_wait_intr+0xf0/0xf0 [ 8424.077943] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8424.098045] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8424.098621] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8424.098628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8424.098631] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8424.098644] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8424.098649] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8424.098653] RAX: 0000000000007f7c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8424.098655] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8424.098657] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8424.098658] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8424.098660] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8424.098664] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8424.098667] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8424.098669] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8424.098678] Call Trace: [ 8424.098684] [ 8424.098689] ? show_regs.cold.9+0x22/0x2f [ 8424.098699] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8424.098706] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8424.098711] ? watchdog+0x40/0x40 [ 8424.098715] ? hrtimer_interrupt+0x128/0x2c0 [ 8424.098720] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8424.098728] ? apic_timer_interrupt+0xf/0x20 [ 8424.098733] [ 8424.098736] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8424.098741] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8424.098747] do_raw_spin_lock+0x177/0x190 [ 8424.098752] _raw_spin_lock+0x21/0x30 [ 8424.098757] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8424.099035] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8424.160322] mdc_reint+0x2d/0x110 [mdc] [ 8424.160372] mdc_unlink+0x658/0x870 [mdc] [ 8424.160398] lmv_unlink+0x34d/0xbd0 [lmv] [ 8424.160428] ll_unlink+0x24d/0x7f0 [lustre] [ 8424.161413] vfs_unlink+0x202/0x330 [ 8424.161423] do_unlinkat+0x2fa/0x460 [ 8424.161429] __x64_sys_unlink+0x37/0x50 [ 8424.161433] do_syscall_64+0xc1/0x440 [ 8424.162198] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8424.162207] RIP: 0033:0x7f83e5f5c10b [ 8424.162215] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8424.162220] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8424.162225] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8424.162306] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8424.162310] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8424.162312] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8424.162315] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8424.516907] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8424.524220] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8424.529282] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8424.535611] ? woken_wake_function+0x30/0x30 [ 8424.536929] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8424.539069] kthread+0x1d1/0x200 [ 8424.540386] ? set_kthread_struct+0x70/0x70 [ 8424.541904] ret_from_fork+0x1f/0x30 [ 8440.060282] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 8440.064665] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8440.117054] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8440.130593] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8440.134958] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8440.142074] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8440.146624] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8440.164444] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8440.169329] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8440.178376] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8440.194686] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8440.207123] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8440.213441] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8440.219268] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8440.223145] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8440.227955] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8440.235729] Call Trace: [ 8440.237816] [ 8440.245465] ? show_regs.cold.9+0x22/0x2f [ 8440.248176] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8440.251535] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8440.254416] ? watchdog+0x40/0x40 [ 8440.255633] ? hrtimer_interrupt+0x128/0x2c0 [ 8440.264143] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8440.271866] ? apic_timer_interrupt+0xf/0x20 [ 8440.279272] [ 8440.279938] ? native_safe_halt+0x2e/0x50 [ 8440.281706] kvm_wait+0xcd/0xe0 [ 8440.286863] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8440.295330] do_raw_spin_lock+0x177/0x190 [ 8440.296657] _raw_spin_lock+0x21/0x30 [ 8440.306361] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8440.310672] process_one_work+0x2c8/0x700 [ 8440.312990] worker_thread+0x296/0x6e0 [ 8440.314857] ? rescuer_thread+0x570/0x570 [ 8440.316790] kthread+0x1d1/0x200 [ 8440.318145] ? set_kthread_struct+0x70/0x70 [ 8440.319971] ret_from_fork+0x1f/0x30 [ 8448.056088] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 8448.060592] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8448.079334] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8448.083743] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8448.088199] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8448.090983] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8448.098517] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8448.101427] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8448.104574] RDX: 0000000000000ed3 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8448.107471] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8448.110552] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8448.113717] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8448.116739] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8448.120316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8448.122831] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8448.126187] Call Trace: [ 8448.127535] [ 8448.128351] ? show_regs.cold.9+0x22/0x2f [ 8448.130157] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8448.132385] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8448.134238] ? watchdog+0x40/0x40 [ 8448.135630] ? hrtimer_interrupt+0x128/0x2c0 [ 8448.137561] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8448.139734] ? apic_timer_interrupt+0xf/0x20 [ 8448.142319] [ 8448.142945] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8448.145847] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8448.147731] do_raw_spin_lock+0x177/0x190 [ 8448.150096] _raw_spin_lock+0x21/0x30 [ 8448.151662] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8448.154752] obd_mod_free+0x46/0x140 [ptlrpc] [ 8448.157732] mdc_commit_open+0x91/0xb0 [mdc] [ 8448.159864] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8448.163921] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8448.167219] after_reply+0xe2c/0x1780 [ptlrpc] [ 8448.169795] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8448.172794] ? __next_timer_interrupt+0x160/0x160 [ 8448.175138] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8448.177841] ? do_wait_intr+0xf0/0xf0 [ 8448.179606] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8448.183193] kthread+0x1d1/0x200 [ 8448.185228] ? set_kthread_struct+0x70/0x70 [ 8448.186741] ret_from_fork+0x1f/0x30 [ 8451.553143] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5938s! [ 8451.556500] Showing busy workqueues and worker pools: [ 8451.558819] workqueue events: flags=0x0 [ 8451.560823] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8451.563762] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8451.569149] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8451.580052] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8451.587950] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8451.594130] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8451.603475] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8451.609624] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8451.623058] workqueue events_long: flags=0x0 [ 8451.624853] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8451.634729] pending: lu_objects_flush [obdclass] [ 8451.642747] workqueue events_unbound: flags=0x2 [ 8451.652417] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8451.660815] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8451.669742] workqueue events_power_efficient: flags=0x80 [ 8451.675720] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8451.681147] pending: neigh_periodic_work, do_cache_clean [ 8451.689709] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8451.704131] pending: neigh_periodic_work, check_lifetime [ 8451.713674] workqueue mm_percpu_wq: flags=0x8 [ 8451.718530] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8451.727069] pending: vmstat_update, lru_add_drain_per_cpu [ 8451.733770] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8451.740826] pending: vmstat_update [ 8451.744737] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8451.750623] pending: lru_add_drain_per_cpu BAR(48) [ 8451.755526] workqueue ipv6_addrconf: flags=0x40008 [ 8451.759059] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8451.764721] pending: addrconf_verify_work [ 8451.768315] workqueue knbd0-recv: flags=0x1a [ 8451.771063] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8451.775557] in-flight: 650:recv_work [ 8451.777602] workqueue ptlrpc_pinger: flags=0x2 [ 8451.782086] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8451.794036] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8451.800749] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5938s workers=4 idle: 36 466 29757 [ 8451.810603] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8441s workers=2 idle: 652 [ 8452.059124] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 8452.062024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8452.063177] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8452.063310] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8452.063317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8452.063319] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8452.063333] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8452.063337] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 8452.063342] RAX: 0000000000002694 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8452.063344] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8452.063346] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8452.063348] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8452.063350] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8452.063353] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8452.063357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8452.063359] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8452.063367] Call Trace: [ 8452.063372] [ 8452.063379] ? show_regs.cold.9+0x22/0x2f [ 8452.063388] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8452.063393] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8452.063397] ? watchdog+0x40/0x40 [ 8452.063401] ? hrtimer_interrupt+0x128/0x2c0 [ 8452.063405] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8452.063419] ? apic_timer_interrupt+0xf/0x20 [ 8452.063424] [ 8452.063537] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8452.066595] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8452.066682] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8452.066689] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8452.066693] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8452.094030] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8452.094055] do_raw_spin_lock+0x177/0x190 [ 8452.094063] _raw_spin_lock+0x21/0x30 [ 8452.094072] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8452.094381] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8452.094580] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8452.104877] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8452.108721] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8452.115165] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8452.115185] RAX: 000000000000601c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8452.115188] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8452.115190] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8452.115192] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8452.115195] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8452.115198] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8452.115201] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8452.115204] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8452.115211] Call Trace: [ 8452.115217] [ 8452.115222] ? show_regs.cold.9+0x22/0x2f [ 8452.115235] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8452.115241] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8452.115245] ? watchdog+0x40/0x40 [ 8452.115249] ? hrtimer_interrupt+0x128/0x2c0 [ 8452.115253] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8452.115261] ? apic_timer_interrupt+0xf/0x20 [ 8452.115265] [ 8452.115268] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8452.115274] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8452.115279] do_raw_spin_lock+0x177/0x190 [ 8452.115283] _raw_spin_lock+0x21/0x30 [ 8452.115289] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8452.126825] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8452.131062] ? do_raw_spin_unlock+0x75/0x190 [ 8452.131096] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8452.134370] ? woken_wake_function+0x30/0x30 [ 8452.134389] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8452.134671] kthread+0x1d1/0x200 [ 8452.134679] ? set_kthread_struct+0x70/0x70 [ 8452.134683] ret_from_fork+0x1f/0x30 [ 8452.315433] ? do_wait_intr+0xf0/0xf0 [ 8452.317174] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8452.319895] mdc_reint+0x2d/0x110 [mdc] [ 8452.321658] mdc_unlink+0x658/0x870 [mdc] [ 8452.323567] lmv_unlink+0x34d/0xbd0 [lmv] [ 8452.325409] ll_unlink+0x24d/0x7f0 [lustre] [ 8452.327434] vfs_unlink+0x202/0x330 [ 8452.328894] do_unlinkat+0x2fa/0x460 [ 8452.330790] __x64_sys_unlink+0x37/0x50 [ 8452.332881] do_syscall_64+0xc1/0x440 [ 8452.334610] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8452.336839] RIP: 0033:0x7f83e5f5c10b [ 8452.338485] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8452.346659] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8452.350254] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8452.352762] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8452.355828] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8452.358995] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8452.362811] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8468.067765] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 8468.077180] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8468.143595] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8468.148221] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8468.155508] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8468.158405] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8468.163871] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8468.185093] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8468.188654] RAX: 0000000000007012 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8468.198380] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8468.201688] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8468.206589] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8468.211738] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8468.217449] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8468.222783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8468.226625] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8468.231469] Call Trace: [ 8468.232690] [ 8468.234580] ? show_regs.cold.9+0x22/0x2f [ 8468.238304] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8468.240686] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8468.248610] ? watchdog+0x40/0x40 [ 8468.250988] ? hrtimer_interrupt+0x128/0x2c0 [ 8468.254424] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8468.257761] ? apic_timer_interrupt+0xf/0x20 [ 8468.265511] [ 8468.266785] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8468.273446] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8468.275943] do_raw_spin_lock+0x177/0x190 [ 8468.279825] _raw_spin_lock+0x21/0x30 [ 8468.281204] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8468.284796] process_one_work+0x2c8/0x700 [ 8468.296098] worker_thread+0x296/0x6e0 [ 8468.298643] ? rescuer_thread+0x570/0x570 [ 8468.307424] kthread+0x1d1/0x200 [ 8468.311685] ? set_kthread_struct+0x70/0x70 [ 8468.322088] ret_from_fork+0x1f/0x30 [ 8476.058489] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 8476.062508] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8476.082465] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8476.088367] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8476.093562] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8476.095730] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8476.104826] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8476.108230] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8476.112135] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8476.115686] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8476.118830] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8476.122251] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8476.126072] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8476.130145] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8476.133339] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8476.137556] Call Trace: [ 8476.138939] [ 8476.140149] ? show_regs.cold.9+0x22/0x2f [ 8476.142325] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8476.144849] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8476.147442] ? watchdog+0x40/0x40 [ 8476.149150] ? hrtimer_interrupt+0x128/0x2c0 [ 8476.151249] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8476.154583] ? apic_timer_interrupt+0xf/0x20 [ 8476.156550] [ 8476.157834] ? native_safe_halt+0x2e/0x50 [ 8476.159956] kvm_wait+0xcd/0xe0 [ 8476.161603] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8476.164345] do_raw_spin_lock+0x177/0x190 [ 8476.166231] _raw_spin_lock+0x21/0x30 [ 8476.168274] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8476.171316] obd_mod_free+0x46/0x140 [ptlrpc] [ 8476.174408] mdc_commit_open+0x91/0xb0 [mdc] [ 8476.176680] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8476.179613] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8476.182965] after_reply+0xe2c/0x1780 [ptlrpc] [ 8476.186217] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8476.189908] ? __next_timer_interrupt+0x160/0x160 [ 8476.192362] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8476.194615] ? do_wait_intr+0xf0/0xf0 [ 8476.196567] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8476.199286] kthread+0x1d1/0x200 [ 8476.200902] ? set_kthread_struct+0x70/0x70 [ 8476.202780] ret_from_fork+0x1f/0x30 [ 8480.059009] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 8480.061169] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8480.062311] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8480.079354] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8480.079369] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8480.079373] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8480.079387] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8480.079390] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8480.079395] RAX: 0000000000002db3 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8480.079398] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8480.079400] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8480.079402] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8480.079405] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8480.079408] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8480.079411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8480.079413] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8480.079421] Call Trace: [ 8480.079427] [ 8480.079433] ? show_regs.cold.9+0x22/0x2f [ 8480.079443] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8480.079449] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8480.079454] ? watchdog+0x40/0x40 [ 8480.079458] ? hrtimer_interrupt+0x128/0x2c0 [ 8480.079462] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8480.079470] ? apic_timer_interrupt+0xf/0x20 [ 8480.079474] [ 8480.079477] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8480.079481] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8480.079488] do_raw_spin_lock+0x177/0x190 [ 8480.079493] _raw_spin_lock+0x21/0x30 [ 8480.079497] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8480.079789] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8480.079988] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8480.080346] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8480.080561] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8480.080852] ? woken_wake_function+0x30/0x30 [ 8480.080861] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8480.081076] kthread+0x1d1/0x200 [ 8480.081083] ? set_kthread_struct+0x70/0x70 [ 8480.081086] ret_from_fork+0x1f/0x30 [ 8480.182045] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8480.201988] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8480.207401] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8480.210552] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8480.213350] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8480.222055] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8480.225224] RAX: 0000000000007463 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8480.228029] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8480.231387] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8480.233723] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8480.236052] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8480.239032] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8480.241513] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8480.243783] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8480.246604] Call Trace: [ 8480.247802] [ 8480.248705] ? show_regs.cold.9+0x22/0x2f [ 8480.250463] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8480.252593] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8480.254792] ? watchdog+0x40/0x40 [ 8480.256427] ? hrtimer_interrupt+0x128/0x2c0 [ 8480.258281] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8480.260243] ? apic_timer_interrupt+0xf/0x20 [ 8480.262252] [ 8480.263078] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8480.265376] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8480.267673] do_raw_spin_lock+0x177/0x190 [ 8480.269730] _raw_spin_lock+0x21/0x30 [ 8480.271341] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8480.273704] ? do_raw_spin_unlock+0x75/0x190 [ 8480.275502] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8480.277983] ? do_wait_intr+0xf0/0xf0 [ 8480.279647] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8480.282248] mdc_reint+0x2d/0x110 [mdc] [ 8480.284642] mdc_unlink+0x658/0x870 [mdc] [ 8480.286514] lmv_unlink+0x34d/0xbd0 [lmv] [ 8480.288384] ll_unlink+0x24d/0x7f0 [lustre] [ 8480.290434] vfs_unlink+0x202/0x330 [ 8480.291782] do_unlinkat+0x2fa/0x460 [ 8480.293587] __x64_sys_unlink+0x37/0x50 [ 8480.295559] do_syscall_64+0xc1/0x440 [ 8480.297318] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8480.300002] RIP: 0033:0x7f83e5f5c10b [ 8480.301719] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8480.311036] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8480.314830] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8480.319008] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8480.322472] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8480.326072] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8480.330138] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8482.282595] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5969s! [ 8482.293405] Showing busy workqueues and worker pools: [ 8482.297285] workqueue events: flags=0x0 [ 8482.299372] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8482.303222] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8482.307782] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8482.312949] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8482.315545] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8482.318270] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8482.321631] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8482.324318] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8482.328212] workqueue events_long: flags=0x0 [ 8482.330434] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8482.335083] pending: lu_objects_flush [obdclass] [ 8482.340606] workqueue events_unbound: flags=0x2 [ 8482.346263] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8482.349567] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8482.352817] workqueue events_power_efficient: flags=0x80 [ 8482.356424] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8482.361884] pending: neigh_periodic_work, do_cache_clean [ 8482.365931] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8482.368643] pending: neigh_periodic_work, check_lifetime [ 8482.375182] workqueue mm_percpu_wq: flags=0x8 [ 8482.382164] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8482.390436] pending: vmstat_update, lru_add_drain_per_cpu [ 8482.393517] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8482.396291] pending: vmstat_update [ 8482.397828] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8482.401343] pending: lru_add_drain_per_cpu BAR(48) [ 8482.412809] workqueue ipv6_addrconf: flags=0x40008 [ 8482.415712] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8482.420777] pending: addrconf_verify_work [ 8482.422161] workqueue knbd0-recv: flags=0x1a [ 8482.423280] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8482.434368] in-flight: 650:recv_work [ 8482.437487] workqueue ptlrpc_pinger: flags=0x2 [ 8482.441192] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8482.445806] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8482.455359] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5969s workers=4 idle: 36 466 29757 [ 8482.459779] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8472s workers=2 idle: 652 [ 8496.062079] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 8496.066822] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8496.125185] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8496.139322] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8496.152708] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8496.163774] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8496.176045] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8496.216659] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8496.224582] RAX: 0000000000001ad3 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8496.237140] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8496.248682] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8496.258509] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8496.268203] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8496.272695] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8496.284055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8496.289078] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8496.299061] Call Trace: [ 8496.301814] [ 8496.302583] ? show_regs.cold.9+0x22/0x2f [ 8496.312843] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8496.316184] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8496.321413] ? watchdog+0x40/0x40 [ 8496.326415] ? hrtimer_interrupt+0x128/0x2c0 [ 8496.329186] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8496.333617] ? apic_timer_interrupt+0xf/0x20 [ 8496.339124] [ 8496.342609] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8496.352675] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8496.355674] do_raw_spin_lock+0x177/0x190 [ 8496.357922] _raw_spin_lock+0x21/0x30 [ 8496.360163] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8496.363770] process_one_work+0x2c8/0x700 [ 8496.365673] worker_thread+0x296/0x6e0 [ 8496.370603] ? rescuer_thread+0x570/0x570 [ 8496.372888] kthread+0x1d1/0x200 [ 8496.374124] ? set_kthread_struct+0x70/0x70 [ 8496.377711] ret_from_fork+0x1f/0x30 [ 8508.059143] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 8508.059720] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 8508.062023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8508.062036] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8508.062103] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8508.062109] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8508.062112] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8508.062192] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8508.062196] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8508.062200] RAX: 000000000000588b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8508.062203] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8508.062205] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8508.062207] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8508.062210] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8508.062213] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8508.062216] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8508.062218] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8508.062226] Call Trace: [ 8508.062231] [ 8508.062237] ? show_regs.cold.9+0x22/0x2f [ 8508.062247] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8508.062254] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8508.062258] ? watchdog+0x40/0x40 [ 8508.062261] ? hrtimer_interrupt+0x128/0x2c0 [ 8508.062265] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8508.062273] ? apic_timer_interrupt+0xf/0x20 [ 8508.062277] [ 8508.062279] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8508.062284] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8508.062288] do_raw_spin_lock+0x177/0x190 [ 8508.062292] _raw_spin_lock+0x21/0x30 [ 8508.062298] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8508.065363] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8508.065448] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8508.065455] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8508.065510] ? do_raw_spin_unlock+0x75/0x190 [ 8508.065519] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8508.065735] ? do_wait_intr+0xf0/0xf0 [ 8508.065754] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8508.065960] mdc_reint+0x2d/0x110 [mdc] [ 8508.066007] mdc_unlink+0x658/0x870 [mdc] [ 8508.066042] lmv_unlink+0x34d/0xbd0 [lmv] [ 8508.065458] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8508.066775] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8508.066779] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8508.066785] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8508.066787] RDX: 000000000000702d RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8508.066789] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8508.066791] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8508.066793] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8508.066796] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8508.066799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8508.066801] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8508.066811] Call Trace: [ 8508.066815] [ 8508.066819] ? show_regs.cold.9+0x22/0x2f [ 8508.066831] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8508.066836] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8508.066840] ? watchdog+0x40/0x40 [ 8508.066844] ? hrtimer_interrupt+0x128/0x2c0 [ 8508.066848] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8508.066857] ? apic_timer_interrupt+0xf/0x20 [ 8508.066862] [ 8508.066865] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8508.066870] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8508.066874] do_raw_spin_lock+0x177/0x190 [ 8508.066879] _raw_spin_lock+0x21/0x30 [ 8508.066071] ll_unlink+0x24d/0x7f0 [lustre] [ 8508.067232] vfs_unlink+0x202/0x330 [ 8508.067241] do_unlinkat+0x2fa/0x460 [ 8508.066886] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8508.067250] __x64_sys_unlink+0x37/0x50 [ 8508.067255] do_syscall_64+0xc1/0x440 [ 8508.067263] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8508.067269] RIP: 0033:0x7f83e5f5c10b [ 8508.067276] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8508.067280] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8508.067285] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8508.067288] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8508.067291] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8508.067293] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8508.067296] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8508.067353] obd_mod_free+0x46/0x140 [ptlrpc] [ 8508.073958] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8508.074032] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8508.074040] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8508.074044] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8508.074058] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8508.074063] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8508.074067] RAX: 000000000000788a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8508.074070] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8508.074072] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8508.074074] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8508.074077] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8508.074080] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8508.074083] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8508.074085] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8508.078687] Call Trace: [ 8508.078695] [ 8508.078700] ? show_regs.cold.9+0x22/0x2f [ 8508.078712] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8508.078725] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8508.078729] ? watchdog+0x40/0x40 [ 8508.078734] ? hrtimer_interrupt+0x128/0x2c0 [ 8508.078737] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8508.078749] ? apic_timer_interrupt+0xf/0x20 [ 8508.078754] [ 8508.078757] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8508.078764] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8508.078771] do_raw_spin_lock+0x177/0x190 [ 8508.078776] _raw_spin_lock+0x21/0x30 [ 8508.078781] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8508.086952] mdc_commit_open+0x91/0xb0 [mdc] [ 8508.087007] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8508.087286] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8508.087688] after_reply+0xe2c/0x1780 [ptlrpc] [ 8508.087906] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8508.088120] ? __next_timer_interrupt+0x160/0x160 [ 8508.088130] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8508.088350] ? do_wait_intr+0xf0/0xf0 [ 8508.088359] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8508.090162] kthread+0x1d1/0x200 [ 8508.090171] ? set_kthread_struct+0x70/0x70 [ 8508.090175] ret_from_fork+0x1f/0x30 [ 8508.909444] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8508.916838] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8508.925101] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8508.937087] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8508.948569] ? woken_wake_function+0x30/0x30 [ 8508.950555] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8508.957893] kthread+0x1d1/0x200 [ 8508.962527] ? set_kthread_struct+0x70/0x70 [ 8508.967195] ret_from_fork+0x1f/0x30 [ 8512.480081] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5999s! [ 8512.483540] Showing busy workqueues and worker pools: [ 8512.486132] workqueue events: flags=0x0 [ 8512.487929] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8512.491559] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8512.495750] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8512.498778] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8512.513719] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8512.519796] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8512.530717] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8512.537789] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8512.549822] workqueue events_long: flags=0x0 [ 8512.557294] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8512.569682] pending: lu_objects_flush [obdclass] [ 8512.576104] workqueue events_unbound: flags=0x2 [ 8512.579137] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8512.582583] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8512.587812] workqueue events_power_efficient: flags=0x80 [ 8512.595192] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8512.606033] pending: neigh_periodic_work, do_cache_clean [ 8512.612158] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8512.619940] pending: neigh_periodic_work, check_lifetime [ 8512.623873] workqueue mm_percpu_wq: flags=0x8 [ 8512.628344] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8512.637927] pending: vmstat_update, lru_add_drain_per_cpu [ 8512.645133] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8512.653429] pending: vmstat_update [ 8512.656985] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8512.670909] pending: lru_add_drain_per_cpu BAR(48) [ 8512.674451] workqueue ipv6_addrconf: flags=0x40008 [ 8512.678777] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8512.684720] pending: addrconf_verify_work [ 8512.688955] workqueue knbd0-recv: flags=0x1a [ 8512.693495] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8512.698458] in-flight: 650:recv_work [ 8512.700393] workqueue ptlrpc_pinger: flags=0x2 [ 8512.706506] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8512.712557] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8512.715944] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5999s workers=4 idle: 36 466 29757 [ 8512.722967] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8502s workers=2 idle: 652 [ 8521.117046] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 8521.120442] rcu: 0-....: (4153065 ticks this GP) idle=30a/1/0x4000000000000000 softirq=128519/128519 fqs=126395 [ 8521.126008] (detected by 1, t=6540158 jiffies, g=262441, q=59130) [ 8521.129437] Sending NMI from CPU 1 to CPUs 0: [ 8521.134506] NMI backtrace for cpu 0 [ 8521.134513] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8521.134515] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8521.134517] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8521.134519] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8521.134521] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 [ 8521.134523] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8521.134525] RDX: 0000000000000699 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8521.134526] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8521.134527] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8521.134528] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8521.134530] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8521.134531] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8521.134532] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8521.134533] Call Trace: [ 8521.134534] [ 8521.134535] ? show_regs.cold.9+0x22/0x2f [ 8521.134536] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8521.134537] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8521.134538] ? nmi_handle+0xbd/0x210 [ 8521.134539] ? default_do_nmi+0x4b/0x1b0 [ 8521.134540] ? do_nmi+0x280/0x380 [ 8521.134541] ? end_repeat_nmi+0x16/0x1f [ 8521.134542] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8521.134544] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8521.134545] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8521.134546] [ 8521.134547] do_raw_spin_lock+0x177/0x190 [ 8521.134548] _raw_spin_lock+0x21/0x30 [ 8521.134549] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8521.134550] obd_mod_free+0x46/0x140 [ptlrpc] [ 8521.134551] mdc_commit_open+0x91/0xb0 [mdc] [ 8521.134552] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8521.134553] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8521.134554] after_reply+0xe2c/0x1780 [ptlrpc] [ 8521.134555] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8521.134556] ? __next_timer_interrupt+0x160/0x160 [ 8521.134557] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8521.134558] ? do_wait_intr+0xf0/0xf0 [ 8521.134559] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8521.134560] kthread+0x1d1/0x200 [ 8521.134561] ? set_kthread_struct+0x70/0x70 [ 8521.134562] ret_from_fork+0x1f/0x30 [ 8521.134620] rcu: rcu_sched kthread starved for 6009345 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 8521.283100] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8521.289200] rcu: RCU grace-period kthread stack dump: [ 8521.292137] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8521.297644] Call Trace: [ 8521.299403] __schedule+0x351/0xcb0 [ 8521.301460] schedule+0xc0/0x180 [ 8521.304545] schedule_timeout+0xb4/0x190 [ 8521.306692] ? do_raw_spin_unlock+0x75/0x190 [ 8521.309562] ? __next_timer_interrupt+0x160/0x160 [ 8521.313994] ? prepare_to_swait_event+0xf3/0x140 [ 8521.316437] rcu_gp_kthread+0x8dd/0x15b0 [ 8521.321721] ? force_qs_rnp+0x2c0/0x2c0 [ 8521.324419] kthread+0x1d1/0x200 [ 8521.326119] ? set_kthread_struct+0x70/0x70 [ 8521.329266] ret_from_fork+0x1f/0x30 [ 8521.332060] rcu: Stack dump where RCU GP kthread last ran: [ 8521.334932] NMI backtrace for cpu 1 [ 8521.338421] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8521.346661] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8521.352250] Call Trace: [ 8521.354073] [ 8521.354971] ? dump_stack+0xbb/0x10e [ 8521.356914] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8521.359517] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8521.362382] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8521.365060] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8521.368199] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8521.370620] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 8521.373404] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [ 8521.375873] ? update_process_times+0x6d/0xc0 [ 8521.378185] ? tick_sched_handle.isra.18+0x27/0x90 [ 8521.381519] ? tick_sched_timer+0x53/0xb0 [ 8521.383759] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8521.386158] ? tick_sched_handle.isra.18+0x90/0x90 [ 8521.388393] ? hrtimer_interrupt+0x128/0x2c0 [ 8521.390747] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8521.393160] ? apic_timer_interrupt+0xf/0x20 [ 8521.395155] [ 8521.396206] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8521.399224] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8521.401668] ? do_raw_spin_lock+0x177/0x190 [ 8521.403778] ? _raw_spin_lock+0x21/0x30 [ 8521.405456] ? ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8521.408783] ? ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8521.412230] ? ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8521.415726] ? ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8521.419761] ? ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8521.423628] ? woken_wake_function+0x30/0x30 [ 8521.426307] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8521.430614] ? kthread+0x1d1/0x200 [ 8521.432688] ? set_kthread_struct+0x70/0x70 [ 8521.436564] ? ret_from_fork+0x1f/0x30 [ 8524.062356] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 8524.074375] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8524.131064] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8524.145199] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8524.151405] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8524.161085] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8524.167163] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8524.196239] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8524.204461] RAX: 000000000000590c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8524.215376] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8524.224386] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8524.227023] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8524.233901] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8524.241275] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8524.246799] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8524.252035] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8524.255762] Call Trace: [ 8524.257761] [ 8524.260338] ? show_regs.cold.9+0x22/0x2f [ 8524.264444] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8524.276538] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8524.280006] ? watchdog+0x40/0x40 [ 8524.282144] ? hrtimer_interrupt+0x128/0x2c0 [ 8524.284305] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8524.289085] ? apic_timer_interrupt+0xf/0x20 [ 8524.292585] [ 8524.294085] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8524.296420] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8524.300098] do_raw_spin_lock+0x177/0x190 [ 8524.303731] _raw_spin_lock+0x21/0x30 [ 8524.309171] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8524.317053] process_one_work+0x2c8/0x700 [ 8524.319905] worker_thread+0x296/0x6e0 [ 8524.326099] ? rescuer_thread+0x570/0x570 [ 8524.328945] kthread+0x1d1/0x200 [ 8524.333014] ? set_kthread_struct+0x70/0x70 [ 8524.335421] ret_from_fork+0x1f/0x30 [ 8536.062021] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8536.064493] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8536.079979] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8536.083903] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8536.087054] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8536.088709] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8536.095622] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8536.098748] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8536.101845] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8536.105049] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8536.108070] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8536.110091] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8536.112869] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8536.115551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8536.118030] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8536.120419] Call Trace: [ 8536.121111] [ 8536.121732] ? show_regs.cold.9+0x22/0x2f [ 8536.123223] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8536.124836] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8536.126552] ? watchdog+0x40/0x40 [ 8536.127687] ? hrtimer_interrupt+0x128/0x2c0 [ 8536.129126] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8536.131098] ? apic_timer_interrupt+0xf/0x20 [ 8536.132367] [ 8536.132943] ? native_safe_halt+0x2e/0x50 [ 8536.134323] kvm_wait+0xcd/0xe0 [ 8536.135331] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8536.136953] do_raw_spin_lock+0x177/0x190 [ 8536.138318] _raw_spin_lock+0x21/0x30 [ 8536.139967] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8536.141938] ? do_raw_spin_unlock+0x75/0x190 [ 8536.143850] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8536.146053] ? do_wait_intr+0xf0/0xf0 [ 8536.148122] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8536.149927] mdc_reint+0x2d/0x110 [mdc] [ 8536.151473] mdc_unlink+0x658/0x870 [mdc] [ 8536.153904] lmv_unlink+0x34d/0xbd0 [lmv] [ 8536.155214] ll_unlink+0x24d/0x7f0 [lustre] [ 8536.156689] vfs_unlink+0x202/0x330 [ 8536.157758] do_unlinkat+0x2fa/0x460 [ 8536.158929] __x64_sys_unlink+0x37/0x50 [ 8536.160587] do_syscall_64+0xc1/0x440 [ 8536.161953] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8536.164161] RIP: 0033:0x7f83e5f5c10b [ 8536.165347] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8536.171819] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8536.174529] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8536.177276] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8536.180208] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8536.183467] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8536.186807] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8543.202042] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6030s! [ 8543.215343] Showing busy workqueues and worker pools: [ 8543.224785] workqueue events: flags=0x0 [ 8543.228023] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8543.243599] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8543.262358] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8543.277527] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8543.290263] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8543.294930] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8543.299954] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8543.303009] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8543.311960] workqueue events_long: flags=0x0 [ 8543.315370] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8543.330243] pending: lu_objects_flush [obdclass] [ 8543.341954] workqueue events_unbound: flags=0x2 [ 8543.358456] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8543.370909] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8543.382212] workqueue events_power_efficient: flags=0x80 [ 8543.389992] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8543.397980] pending: neigh_periodic_work, do_cache_clean [ 8543.415547] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8543.441211] pending: neigh_periodic_work, check_lifetime [ 8543.462356] workqueue mm_percpu_wq: flags=0x8 [ 8543.482842] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8543.500072] pending: vmstat_update, lru_add_drain_per_cpu [ 8543.512106] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8543.536005] pending: vmstat_update [ 8543.544269] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8543.565824] pending: lru_add_drain_per_cpu BAR(48) [ 8543.570839] workqueue ipv6_addrconf: flags=0x40008 [ 8543.573711] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8543.575932] pending: addrconf_verify_work [ 8543.582490] workqueue knbd0-recv: flags=0x1a [ 8543.593509] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8543.607060] in-flight: 650:recv_work [ 8543.610178] workqueue ptlrpc_pinger: flags=0x2 [ 8543.614160] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8543.628260] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8543.644705] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6030s workers=4 idle: 36 466 29757 [ 8543.649109] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8533s workers=2 idle: 652 [ 8548.060156] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 8548.073938] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8548.105468] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8548.111912] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8548.122013] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8548.125023] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8548.136821] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8548.151553] RAX: 0000000000005301 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8548.159356] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8548.164521] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8548.170591] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8548.173942] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8548.183013] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8548.189422] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8548.192483] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8548.195697] Call Trace: [ 8548.196888] [ 8548.197827] ? show_regs.cold.9+0x22/0x2f [ 8548.200470] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8548.202761] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8548.205635] ? watchdog+0x40/0x40 [ 8548.210132] ? hrtimer_interrupt+0x128/0x2c0 [ 8548.211685] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8548.213158] ? apic_timer_interrupt+0xf/0x20 [ 8548.214172] [ 8548.214671] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8548.217615] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8548.220091] do_raw_spin_lock+0x177/0x190 [ 8548.221834] _raw_spin_lock+0x21/0x30 [ 8548.223744] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8548.226622] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8548.231663] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8548.234088] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8548.237000] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8548.240059] ? woken_wake_function+0x30/0x30 [ 8548.244012] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8548.250070] kthread+0x1d1/0x200 [ 8548.252696] ? set_kthread_struct+0x70/0x70 [ 8548.255304] ret_from_fork+0x1f/0x30 [ 8552.060026] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 8552.068704] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8552.141318] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8552.155354] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8552.166728] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8552.170616] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8552.180758] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8552.203535] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8552.214598] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8552.225636] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8552.232655] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8552.239649] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8552.246616] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8552.254439] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8552.259538] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8552.263566] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8552.269861] Call Trace: [ 8552.271656] [ 8552.275397] ? show_regs.cold.9+0x22/0x2f [ 8552.276690] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8552.290004] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8552.293320] ? watchdog+0x40/0x40 [ 8552.295598] ? hrtimer_interrupt+0x128/0x2c0 [ 8552.299634] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8552.308437] ? apic_timer_interrupt+0xf/0x20 [ 8552.310157] [ 8552.310937] ? native_safe_halt+0x2e/0x50 [ 8552.315914] kvm_wait+0xcd/0xe0 [ 8552.319503] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8552.322650] do_raw_spin_lock+0x177/0x190 [ 8552.325531] _raw_spin_lock+0x21/0x30 [ 8552.326818] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8552.331959] process_one_work+0x2c8/0x700 [ 8552.334964] worker_thread+0x296/0x6e0 [ 8552.338029] ? rescuer_thread+0x570/0x570 [ 8552.340619] kthread+0x1d1/0x200 [ 8552.342818] ? set_kthread_struct+0x70/0x70 [ 8552.347249] ret_from_fork+0x1f/0x30 [ 8564.062126] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8564.065825] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8564.089921] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8564.095212] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8564.099157] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8564.101526] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8564.110361] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8564.114575] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8564.118058] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8564.121831] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8564.126606] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8564.129648] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8564.133369] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8564.137049] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8564.139698] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8564.142791] Call Trace: [ 8564.144359] [ 8564.145304] ? show_regs.cold.9+0x22/0x2f [ 8564.147150] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8564.149385] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8564.151320] ? watchdog+0x40/0x40 [ 8564.152864] ? hrtimer_interrupt+0x128/0x2c0 [ 8564.155234] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8564.158552] ? apic_timer_interrupt+0xf/0x20 [ 8564.161632] [ 8564.162604] ? native_safe_halt+0x2e/0x50 [ 8564.164783] kvm_wait+0xcd/0xe0 [ 8564.166804] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8564.171006] do_raw_spin_lock+0x177/0x190 [ 8564.172574] _raw_spin_lock+0x21/0x30 [ 8564.174006] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8564.176774] ? do_raw_spin_unlock+0x75/0x190 [ 8564.178960] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8564.180863] ? do_wait_intr+0xf0/0xf0 [ 8564.182244] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8564.184715] mdc_reint+0x2d/0x110 [mdc] [ 8564.186634] mdc_unlink+0x658/0x870 [mdc] [ 8564.188974] lmv_unlink+0x34d/0xbd0 [lmv] [ 8564.191537] ll_unlink+0x24d/0x7f0 [lustre] [ 8564.194132] vfs_unlink+0x202/0x330 [ 8564.196295] do_unlinkat+0x2fa/0x460 [ 8564.198236] __x64_sys_unlink+0x37/0x50 [ 8564.201066] do_syscall_64+0xc1/0x440 [ 8564.203270] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8564.206146] RIP: 0033:0x7f83e5f5c10b [ 8564.207720] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8564.218180] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8564.223040] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8564.228503] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8564.231815] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8564.238554] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8564.242747] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8568.056630] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 8568.060898] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8568.110614] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8568.124411] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8568.134020] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8568.137213] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8568.153548] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8568.160892] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8568.166571] RDX: 0000000000004ee0 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8568.176368] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8568.182138] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8568.190117] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8568.195988] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8568.208210] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8568.217150] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8568.231822] Call Trace: [ 8568.234148] [ 8568.239205] ? show_regs.cold.9+0x22/0x2f [ 8568.248072] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8568.259546] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8568.269687] ? watchdog+0x40/0x40 [ 8568.273154] ? hrtimer_interrupt+0x128/0x2c0 [ 8568.277733] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8568.287211] ? apic_timer_interrupt+0xf/0x20 [ 8568.293577] [ 8568.295752] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8568.297664] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8568.305618] do_raw_spin_lock+0x177/0x190 [ 8568.310176] _raw_spin_lock+0x21/0x30 [ 8568.315390] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8568.319652] obd_mod_free+0x46/0x140 [ptlrpc] [ 8568.323357] mdc_commit_open+0x91/0xb0 [mdc] [ 8568.327259] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8568.330101] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8568.334292] after_reply+0xe2c/0x1780 [ptlrpc] [ 8568.339646] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8568.344254] ? __next_timer_interrupt+0x160/0x160 [ 8568.347271] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8568.352177] ? do_wait_intr+0xf0/0xf0 [ 8568.359699] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8568.362673] kthread+0x1d1/0x200 [ 8568.370296] ? set_kthread_struct+0x70/0x70 [ 8568.375616] ret_from_fork+0x1f/0x30 [ 8573.922147] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6060s! [ 8573.924323] Showing busy workqueues and worker pools: [ 8573.925410] workqueue events: flags=0x0 [ 8573.929116] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8573.932324] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8573.937752] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8573.940732] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8573.946605] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8573.951169] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8573.955663] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8573.959197] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8573.966316] workqueue events_long: flags=0x0 [ 8573.968191] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8573.970695] pending: lu_objects_flush [obdclass] [ 8573.972941] workqueue events_unbound: flags=0x2 [ 8573.974687] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8573.977177] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8573.980434] workqueue events_power_efficient: flags=0x80 [ 8573.983646] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8573.989365] pending: neigh_periodic_work, do_cache_clean [ 8573.992135] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8573.996584] pending: neigh_periodic_work, check_lifetime [ 8574.000214] workqueue mm_percpu_wq: flags=0x8 [ 8574.002505] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8574.005867] pending: vmstat_update, lru_add_drain_per_cpu [ 8574.008232] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8574.011651] pending: vmstat_update [ 8574.013214] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8574.016745] pending: lru_add_drain_per_cpu BAR(48) [ 8574.018034] workqueue ipv6_addrconf: flags=0x40008 [ 8574.021218] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8574.023910] pending: addrconf_verify_work [ 8574.026152] workqueue knbd0-recv: flags=0x1a [ 8574.029740] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8574.033019] in-flight: 650:recv_work [ 8574.035556] workqueue ptlrpc_pinger: flags=0x2 [ 8574.036981] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8574.040241] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8574.043277] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6060s workers=4 idle: 36 466 29757 [ 8574.045728] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8564s workers=2 idle: 652 [ 8576.059176] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 8576.065456] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8576.095154] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8576.100260] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8576.104228] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8576.107302] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8576.116296] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 8576.119855] RAX: 0000000000007f92 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8576.123494] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8576.127206] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8576.131285] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8576.134900] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8576.138380] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8576.142842] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8576.145726] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8576.149461] Call Trace: [ 8576.151871] [ 8576.152920] ? show_regs.cold.9+0x22/0x2f [ 8576.156068] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8576.158411] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8576.160915] ? watchdog+0x40/0x40 [ 8576.162788] ? hrtimer_interrupt+0x128/0x2c0 [ 8576.166456] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8576.168639] ? apic_timer_interrupt+0xf/0x20 [ 8576.173212] [ 8576.174894] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8576.180061] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8576.184394] do_raw_spin_lock+0x177/0x190 [ 8576.186446] _raw_spin_lock+0x21/0x30 [ 8576.188342] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8576.192132] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8576.195324] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8576.198318] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8576.201398] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8576.204276] ? woken_wake_function+0x30/0x30 [ 8576.206872] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8576.210290] kthread+0x1d1/0x200 [ 8576.212364] ? set_kthread_struct+0x70/0x70 [ 8576.215894] ret_from_fork+0x1f/0x30 [ 8580.063273] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 8580.069558] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8580.133147] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8580.137936] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8580.144440] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8580.149583] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8580.152622] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8580.162179] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8580.166828] RAX: 0000000000002689 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8580.170595] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8580.175983] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8580.184693] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8580.210639] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8580.247719] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8580.252760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8580.262386] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8580.270192] Call Trace: [ 8580.274003] [ 8580.275767] ? show_regs.cold.9+0x22/0x2f [ 8580.281233] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8580.286907] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8580.292460] ? watchdog+0x40/0x40 [ 8580.293604] ? hrtimer_interrupt+0x128/0x2c0 [ 8580.299048] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8580.302179] ? apic_timer_interrupt+0xf/0x20 [ 8580.310293] [ 8580.312378] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8580.317401] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8580.321100] do_raw_spin_lock+0x177/0x190 [ 8580.325502] _raw_spin_lock+0x21/0x30 [ 8580.331264] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8580.347390] process_one_work+0x2c8/0x700 [ 8580.354553] worker_thread+0x296/0x6e0 [ 8580.360226] ? rescuer_thread+0x570/0x570 [ 8580.364390] kthread+0x1d1/0x200 [ 8580.365615] ? set_kthread_struct+0x70/0x70 [ 8580.370875] ret_from_fork+0x1f/0x30 [ 8592.064058] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8592.079375] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8592.134147] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8592.143220] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8592.153368] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8592.160240] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8592.175846] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8592.185856] RAX: 0000000000003483 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8592.196431] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8592.202590] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8592.214518] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8592.218361] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8592.233455] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8592.249097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8592.259102] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8592.266865] Call Trace: [ 8592.268767] [ 8592.269960] ? show_regs.cold.9+0x22/0x2f [ 8592.273455] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8592.276230] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8592.279223] ? watchdog+0x40/0x40 [ 8592.280942] ? hrtimer_interrupt+0x128/0x2c0 [ 8592.285960] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8592.293394] ? apic_timer_interrupt+0xf/0x20 [ 8592.296255] [ 8592.298302] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8592.301946] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8592.305931] do_raw_spin_lock+0x177/0x190 [ 8592.309026] _raw_spin_lock+0x21/0x30 [ 8592.315181] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8592.318681] ? do_raw_spin_unlock+0x75/0x190 [ 8592.321911] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8592.325762] ? do_wait_intr+0xf0/0xf0 [ 8592.327801] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8592.338078] mdc_reint+0x2d/0x110 [mdc] [ 8592.341637] mdc_unlink+0x658/0x870 [mdc] [ 8592.343952] lmv_unlink+0x34d/0xbd0 [lmv] [ 8592.349776] ll_unlink+0x24d/0x7f0 [lustre] [ 8592.353296] vfs_unlink+0x202/0x330 [ 8592.355437] do_unlinkat+0x2fa/0x460 [ 8592.358406] __x64_sys_unlink+0x37/0x50 [ 8592.360558] do_syscall_64+0xc1/0x440 [ 8592.365096] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8592.372880] RIP: 0033:0x7f83e5f5c10b [ 8592.374782] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8592.399764] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8592.412457] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8592.422410] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8592.429237] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8592.438332] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8592.447886] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8600.056019] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 8600.059284] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8600.083275] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8600.089706] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8600.094098] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8600.096647] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8600.105912] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8600.109570] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8600.112330] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8600.115608] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8600.119666] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8600.123293] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8600.127079] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8600.131254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8600.134582] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8600.138157] Call Trace: [ 8600.139351] [ 8600.140666] ? show_regs.cold.9+0x22/0x2f [ 8600.143540] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8600.146077] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8600.148256] ? watchdog+0x40/0x40 [ 8600.150204] ? hrtimer_interrupt+0x128/0x2c0 [ 8600.152229] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8600.155148] ? apic_timer_interrupt+0xf/0x20 [ 8600.157226] [ 8600.158236] ? native_safe_halt+0x2e/0x50 [ 8600.159915] kvm_wait+0xcd/0xe0 [ 8600.161773] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8600.164267] do_raw_spin_lock+0x177/0x190 [ 8600.166554] _raw_spin_lock+0x21/0x30 [ 8600.168237] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8600.170788] obd_mod_free+0x46/0x140 [ptlrpc] [ 8600.176178] mdc_commit_open+0x91/0xb0 [mdc] [ 8600.178333] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8600.181212] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8600.184732] after_reply+0xe2c/0x1780 [ptlrpc] [ 8600.187706] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8600.190914] ? __next_timer_interrupt+0x160/0x160 [ 8600.193213] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8600.196123] ? do_wait_intr+0xf0/0xf0 [ 8600.198080] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8600.201200] kthread+0x1d1/0x200 [ 8600.203539] ? set_kthread_struct+0x70/0x70 [ 8600.205832] ret_from_fork+0x1f/0x30 [ 8604.059015] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 8604.063017] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8604.094582] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8604.102545] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8604.108379] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8604.112099] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8604.125129] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8604.131883] RAX: 0000000000000196 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8604.136610] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8604.142143] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8604.147505] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8604.151119] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8604.153807] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8604.157623] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8604.160430] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8604.165001] Call Trace: [ 8604.165570] [ 8604.168349] ? show_regs.cold.9+0x22/0x2f [ 8604.169786] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8604.171167] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8604.173400] ? watchdog+0x40/0x40 [ 8604.175389] ? hrtimer_interrupt+0x128/0x2c0 [ 8604.176380] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8604.179045] ? apic_timer_interrupt+0xf/0x20 [ 8604.181958] [ 8604.182419] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8604.184940] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8604.187661] do_raw_spin_lock+0x177/0x190 [ 8604.189341] _raw_spin_lock+0x21/0x30 [ 8604.190375] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8604.192472] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8604.194048] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8604.195446] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8604.197060] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8604.198423] ? woken_wake_function+0x30/0x30 [ 8604.199419] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8604.201037] kthread+0x1d1/0x200 [ 8604.202002] ? set_kthread_struct+0x70/0x70 [ 8604.203399] ret_from_fork+0x1f/0x30 [ 8604.205133] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6091s! [ 8604.210120] Showing busy workqueues and worker pools: [ 8604.213684] workqueue events: flags=0x0 [ 8604.216300] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8604.223145] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8604.226550] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8604.230314] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8604.234012] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8604.237238] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8604.240888] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8604.243767] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8604.247772] workqueue events_long: flags=0x0 [ 8604.249415] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8604.252525] pending: lu_objects_flush [obdclass] [ 8604.255362] workqueue events_unbound: flags=0x2 [ 8604.258752] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8604.262880] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8604.265759] workqueue events_power_efficient: flags=0x80 [ 8604.268253] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8604.271994] pending: neigh_periodic_work, do_cache_clean [ 8604.275007] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8604.278788] pending: neigh_periodic_work, check_lifetime [ 8604.281972] workqueue mm_percpu_wq: flags=0x8 [ 8604.284174] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8604.287298] pending: vmstat_update, lru_add_drain_per_cpu [ 8604.290316] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8604.294010] pending: vmstat_update [ 8604.296009] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8604.298964] pending: lru_add_drain_per_cpu BAR(48) [ 8604.301337] workqueue ipv6_addrconf: flags=0x40008 [ 8604.303741] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8604.307240] pending: addrconf_verify_work [ 8604.313293] workqueue knbd0-recv: flags=0x1a [ 8604.315645] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8604.322699] in-flight: 650:recv_work [ 8604.325159] workqueue ptlrpc_pinger: flags=0x2 [ 8604.327825] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8604.331212] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8604.334525] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6091s workers=4 idle: 36 466 29757 [ 8604.338439] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8594s workers=2 idle: 652 [ 8608.060030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 8608.063470] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8608.084130] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8608.089041] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8608.092816] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8608.096082] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8608.098139] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8608.106360] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8608.109399] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8608.112495] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8608.115541] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8608.118535] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8608.121711] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8608.124699] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8608.128160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8608.130926] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8608.133885] Call Trace: [ 8608.134548] [ 8608.135098] ? show_regs.cold.9+0x22/0x2f [ 8608.136765] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8608.138762] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8608.140747] ? watchdog+0x40/0x40 [ 8608.142161] ? hrtimer_interrupt+0x128/0x2c0 [ 8608.144144] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8608.146189] ? apic_timer_interrupt+0xf/0x20 [ 8608.147933] [ 8608.148885] ? native_safe_halt+0x2e/0x50 [ 8608.150375] kvm_wait+0xcd/0xe0 [ 8608.151622] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8608.153930] do_raw_spin_lock+0x177/0x190 [ 8608.155952] _raw_spin_lock+0x21/0x30 [ 8608.157608] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8608.160109] process_one_work+0x2c8/0x700 [ 8608.162237] worker_thread+0x296/0x6e0 [ 8608.163861] ? rescuer_thread+0x570/0x570 [ 8608.166211] kthread+0x1d1/0x200 [ 8608.167481] ? set_kthread_struct+0x70/0x70 [ 8608.169504] ret_from_fork+0x1f/0x30 [ 8620.062025] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 8620.064959] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8620.085488] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8620.094873] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8620.101686] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8620.109128] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8620.124365] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8620.129049] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8620.132277] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8620.135510] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8620.138725] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8620.141320] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8620.143930] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8620.146965] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8620.149158] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8620.151977] Call Trace: [ 8620.153035] [ 8620.153790] ? show_regs.cold.9+0x22/0x2f [ 8620.155535] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8620.161154] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8620.163944] ? watchdog+0x40/0x40 [ 8620.165737] ? hrtimer_interrupt+0x128/0x2c0 [ 8620.167531] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8620.169221] ? apic_timer_interrupt+0xf/0x20 [ 8620.170962] [ 8620.171594] ? native_safe_halt+0x2e/0x50 [ 8620.173010] kvm_wait+0xcd/0xe0 [ 8620.174737] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8620.177998] do_raw_spin_lock+0x177/0x190 [ 8620.179440] _raw_spin_lock+0x21/0x30 [ 8620.180904] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8620.183119] ? do_raw_spin_unlock+0x75/0x190 [ 8620.185007] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8620.186999] ? do_wait_intr+0xf0/0xf0 [ 8620.188375] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8620.190553] mdc_reint+0x2d/0x110 [mdc] [ 8620.192682] mdc_unlink+0x658/0x870 [mdc] [ 8620.194225] lmv_unlink+0x34d/0xbd0 [lmv] [ 8620.195856] ll_unlink+0x24d/0x7f0 [lustre] [ 8620.197774] vfs_unlink+0x202/0x330 [ 8620.199165] do_unlinkat+0x2fa/0x460 [ 8620.200775] __x64_sys_unlink+0x37/0x50 [ 8620.202320] do_syscall_64+0xc1/0x440 [ 8620.203976] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8620.206097] RIP: 0033:0x7f83e5f5c10b [ 8620.207379] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8620.214556] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8620.217768] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8620.220971] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8620.224256] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8620.227324] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8620.230453] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8632.056028] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 8632.065632] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8632.066409] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 8632.085732] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8632.085747] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8632.085750] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8632.085770] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8632.085775] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8632.085781] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8632.085784] RDX: 00000000000066ee RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8632.085787] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8632.085789] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8632.085792] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8632.085795] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8632.085798] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8632.085801] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8632.085808] Call Trace: [ 8632.085814] [ 8632.085821] ? show_regs.cold.9+0x22/0x2f [ 8632.085832] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8632.085839] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8632.085843] ? watchdog+0x40/0x40 [ 8632.085848] ? hrtimer_interrupt+0x128/0x2c0 [ 8632.085852] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8632.085860] ? apic_timer_interrupt+0xf/0x20 [ 8632.085865] [ 8632.085868] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8632.085873] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8632.085878] do_raw_spin_lock+0x177/0x190 [ 8632.085884] _raw_spin_lock+0x21/0x30 [ 8632.085891] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8632.086234] obd_mod_free+0x46/0x140 [ptlrpc] [ 8632.086576] mdc_commit_open+0x91/0xb0 [mdc] [ 8632.086618] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8632.086823] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8632.087115] after_reply+0xe2c/0x1780 [ptlrpc] [ 8632.087524] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8632.087722] ? __next_timer_interrupt+0x160/0x160 [ 8632.087731] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8632.088379] ? do_wait_intr+0xf0/0xf0 [ 8632.088392] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8632.088674] kthread+0x1d1/0x200 [ 8632.088681] ? set_kthread_struct+0x70/0x70 [ 8632.088684] ret_from_fork+0x1f/0x30 [ 8632.219013] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8632.239730] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8632.245100] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8632.248708] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8632.250867] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8632.257561] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8632.262034] RAX: 0000000000007ba3 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8632.265658] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8632.268910] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8632.271637] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8632.274671] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8632.277236] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8632.280480] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8632.285518] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8632.288215] Call Trace: [ 8632.289022] [ 8632.289894] ? show_regs.cold.9+0x22/0x2f [ 8632.291808] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8632.293581] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8632.295180] ? watchdog+0x40/0x40 [ 8632.296667] ? hrtimer_interrupt+0x128/0x2c0 [ 8632.298661] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8632.301154] ? apic_timer_interrupt+0xf/0x20 [ 8632.303102] [ 8632.304033] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8632.305563] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8632.307152] do_raw_spin_lock+0x177/0x190 [ 8632.308672] _raw_spin_lock+0x21/0x30 [ 8632.309862] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8632.312304] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8632.314483] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8632.316920] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8632.319466] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8632.321832] ? woken_wake_function+0x30/0x30 [ 8632.323561] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8632.326668] kthread+0x1d1/0x200 [ 8632.327923] ? set_kthread_struct+0x70/0x70 [ 8632.329828] ret_from_fork+0x1f/0x30 [ 8634.848125] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6121s! [ 8634.851924] Showing busy workqueues and worker pools: [ 8634.853570] workqueue events: flags=0x0 [ 8634.854725] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8634.856707] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8634.860461] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8634.862693] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8634.865383] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8634.870304] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8634.872742] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8634.878808] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8634.883649] workqueue events_long: flags=0x0 [ 8634.885823] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8634.889810] pending: lu_objects_flush [obdclass] [ 8634.892012] workqueue events_unbound: flags=0x2 [ 8634.898987] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8634.903639] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8634.906519] workqueue events_power_efficient: flags=0x80 [ 8634.910131] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8634.916602] pending: neigh_periodic_work, do_cache_clean [ 8634.920708] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8634.929000] pending: neigh_periodic_work, check_lifetime [ 8634.932558] workqueue mm_percpu_wq: flags=0x8 [ 8634.938153] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8634.946710] pending: vmstat_update, lru_add_drain_per_cpu [ 8634.953214] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8634.962219] pending: vmstat_update [ 8634.967054] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8634.971048] pending: lru_add_drain_per_cpu BAR(48) [ 8634.975231] workqueue ipv6_addrconf: flags=0x40008 [ 8634.979325] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8634.987160] pending: addrconf_verify_work [ 8634.990096] workqueue knbd0-recv: flags=0x1a [ 8634.993735] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8634.999795] in-flight: 650:recv_work [ 8635.003188] workqueue ptlrpc_pinger: flags=0x2 [ 8635.007486] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8635.014161] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8635.019564] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6121s workers=4 idle: 36 466 29757 [ 8635.025721] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8625s workers=2 idle: 652 [ 8636.060013] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 8636.063657] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8636.085791] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8636.090728] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8636.094889] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8636.098750] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8636.101847] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8636.109824] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8636.113745] RAX: 00000000000022a5 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8636.117290] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8636.121687] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8636.125202] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8636.128672] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8636.131150] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8636.133824] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8636.136005] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8636.138594] Call Trace: [ 8636.139632] [ 8636.140638] ? show_regs.cold.9+0x22/0x2f [ 8636.142563] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8636.144825] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8636.146515] ? watchdog+0x40/0x40 [ 8636.147553] ? hrtimer_interrupt+0x128/0x2c0 [ 8636.150157] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8636.151616] ? apic_timer_interrupt+0xf/0x20 [ 8636.153652] [ 8636.154230] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8636.155690] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8636.157465] do_raw_spin_lock+0x177/0x190 [ 8636.158572] _raw_spin_lock+0x21/0x30 [ 8636.159617] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8636.161441] process_one_work+0x2c8/0x700 [ 8636.162598] worker_thread+0x296/0x6e0 [ 8636.163697] ? rescuer_thread+0x570/0x570 [ 8636.165012] kthread+0x1d1/0x200 [ 8636.165992] ? set_kthread_struct+0x70/0x70 [ 8636.167188] ret_from_fork+0x1f/0x30 [ 8648.064277] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 8648.068954] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8648.120663] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8648.133914] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8648.138484] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8648.140807] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8648.152078] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8648.154840] RAX: 0000000000004e21 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8648.157103] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8648.159466] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8648.161922] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8648.164701] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8648.167464] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8648.169870] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8648.171943] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8648.174124] Call Trace: [ 8648.174998] [ 8648.175718] ? show_regs.cold.9+0x22/0x2f [ 8648.177074] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8648.180726] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8648.182737] ? watchdog+0x40/0x40 [ 8648.184486] ? hrtimer_interrupt+0x128/0x2c0 [ 8648.185954] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8648.187719] ? apic_timer_interrupt+0xf/0x20 [ 8648.189648] [ 8648.192955] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8648.196640] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8648.199134] do_raw_spin_lock+0x177/0x190 [ 8648.202631] _raw_spin_lock+0x21/0x30 [ 8648.204662] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8648.219401] ? do_raw_spin_unlock+0x75/0x190 [ 8648.220454] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8648.221849] ? do_wait_intr+0xf0/0xf0 [ 8648.228130] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8648.232691] mdc_reint+0x2d/0x110 [mdc] [ 8648.235605] mdc_unlink+0x658/0x870 [mdc] [ 8648.244950] lmv_unlink+0x34d/0xbd0 [lmv] [ 8648.252297] ll_unlink+0x24d/0x7f0 [lustre] [ 8648.255188] vfs_unlink+0x202/0x330 [ 8648.260667] do_unlinkat+0x2fa/0x460 [ 8648.264365] __x64_sys_unlink+0x37/0x50 [ 8648.267581] do_syscall_64+0xc1/0x440 [ 8648.268908] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8648.271854] RIP: 0033:0x7f83e5f5c10b [ 8648.274602] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8648.282643] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8648.285355] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8648.288385] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8648.291509] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8648.303252] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8648.310168] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8660.056124] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 8660.059610] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 8660.059623] Modules linked in: [ 8660.063292] Modules linked in: [ 8660.063296] lustre(O) osp(O) ofd(O) [ 8660.069280] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) [ 8660.070889] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8660.070982] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8660.070989] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8660.070992] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8660.071017] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8660.071022] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 8660.071027] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8660.071030] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8660.071032] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8660.071034] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8660.071037] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8660.071039] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8660.071042] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8660.071045] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8660.071058] Call Trace: [ 8660.071064] [ 8660.071070] ? show_regs.cold.9+0x22/0x2f [ 8660.071079] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8660.071085] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8660.071095] ? watchdog+0x40/0x40 [ 8660.071100] ? hrtimer_interrupt+0x128/0x2c0 [ 8660.071104] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8660.071111] ? apic_timer_interrupt+0xf/0x20 [ 8660.071116] [ 8660.071119] ? native_safe_halt+0x2e/0x50 [ 8660.071122] kvm_wait+0xcd/0xe0 [ 8660.071131] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8660.071139] do_raw_spin_lock+0x177/0x190 [ 8660.071143] _raw_spin_lock+0x21/0x30 [ 8660.071148] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8660.081213] ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8660.081264] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8660.081270] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8660.081273] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8660.081286] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8660.081290] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8660.087874] obd_mod_free+0x46/0x140 [ptlrpc] [ 8660.088345] mdc_commit_open+0x91/0xb0 [mdc] [ 8660.088383] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8660.088695] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8660.088907] after_reply+0xe2c/0x1780 [ptlrpc] [ 8660.089336] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8660.089532] ? __next_timer_interrupt+0x160/0x160 [ 8660.089543] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8660.089760] ? do_wait_intr+0xf0/0xf0 [ 8660.089770] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8660.090124] kthread+0x1d1/0x200 [ 8660.090135] ? set_kthread_struct+0x70/0x70 [ 8660.090140] ret_from_fork+0x1f/0x30 [ 8660.558146] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8660.565936] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8660.573689] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8660.582081] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8660.593629] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8660.606352] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8660.619456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8660.627013] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8660.635747] Call Trace: [ 8660.640042] [ 8660.640920] ? show_regs.cold.9+0x22/0x2f [ 8660.646881] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8660.649183] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8660.655345] ? watchdog+0x40/0x40 [ 8660.662278] ? hrtimer_interrupt+0x128/0x2c0 [ 8660.667081] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8660.669103] ? apic_timer_interrupt+0xf/0x20 [ 8660.674706] [ 8660.675772] ? native_safe_halt+0x2e/0x50 [ 8660.683870] kvm_wait+0xcd/0xe0 [ 8660.686025] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8660.692479] do_raw_spin_lock+0x177/0x190 [ 8660.695660] _raw_spin_lock+0x21/0x30 [ 8660.699333] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8660.708843] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8660.714881] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8660.719581] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8660.723331] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8660.727742] ? woken_wake_function+0x30/0x30 [ 8660.729549] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8660.732296] kthread+0x1d1/0x200 [ 8660.733821] ? set_kthread_struct+0x70/0x70 [ 8660.735846] ret_from_fork+0x1f/0x30 [ 8664.062130] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 8664.073809] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8664.132237] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8664.142876] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8664.155581] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8664.162503] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8664.167644] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8664.184272] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8664.188387] RAX: 0000000000007083 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8664.191379] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8664.197248] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8664.200283] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8664.203825] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8664.207162] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8664.211478] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8664.214000] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8664.217459] Call Trace: [ 8664.218628] [ 8664.219524] ? show_regs.cold.9+0x22/0x2f [ 8664.221700] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8664.227640] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8664.229375] ? watchdog+0x40/0x40 [ 8664.230500] ? hrtimer_interrupt+0x128/0x2c0 [ 8664.231595] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8664.239637] ? apic_timer_interrupt+0xf/0x20 [ 8664.242312] [ 8664.243454] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8664.252023] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8664.255704] do_raw_spin_lock+0x177/0x190 [ 8664.260159] _raw_spin_lock+0x21/0x30 [ 8664.262835] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8664.265716] process_one_work+0x2c8/0x700 [ 8664.268035] worker_thread+0x296/0x6e0 [ 8664.269585] ? rescuer_thread+0x570/0x570 [ 8664.271529] kthread+0x1d1/0x200 [ 8664.277307] ? set_kthread_struct+0x70/0x70 [ 8664.278551] ret_from_fork+0x1f/0x30 [ 8665.056109] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6151s! [ 8665.064890] Showing busy workqueues and worker pools: [ 8665.073974] workqueue events: flags=0x0 [ 8665.075225] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8665.087490] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8665.100018] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8665.111993] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8665.122049] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8665.132074] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8665.139354] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8665.145374] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8665.157624] workqueue events_long: flags=0x0 [ 8665.167337] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8665.173309] pending: lu_objects_flush [obdclass] [ 8665.175753] workqueue events_unbound: flags=0x2 [ 8665.177712] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8665.179546] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8665.180921] workqueue events_power_efficient: flags=0x80 [ 8665.182624] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8665.186280] pending: neigh_periodic_work, do_cache_clean [ 8665.189796] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8665.192833] pending: neigh_periodic_work, check_lifetime [ 8665.202693] workqueue mm_percpu_wq: flags=0x8 [ 8665.207513] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8665.214240] pending: vmstat_update, lru_add_drain_per_cpu [ 8665.221600] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8665.232644] pending: vmstat_update [ 8665.237307] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8665.244253] pending: lru_add_drain_per_cpu BAR(48) [ 8665.253323] workqueue ipv6_addrconf: flags=0x40008 [ 8665.257882] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8665.266325] pending: addrconf_verify_work [ 8665.268830] workqueue knbd0-recv: flags=0x1a [ 8665.270855] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8665.276859] in-flight: 650:recv_work [ 8665.278351] workqueue ptlrpc_pinger: flags=0x2 [ 8665.280482] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8665.282618] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8665.284808] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6152s workers=4 idle: 36 466 29757 [ 8665.287404] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8655s workers=2 idle: 652 [ 8676.065775] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 8676.078453] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8676.123727] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8676.136295] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8676.143556] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8676.148877] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8676.168652] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8676.179784] RAX: 0000000000007407 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8676.189339] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8676.196715] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8676.204976] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8676.212448] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8676.220595] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8676.228601] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8676.236907] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8676.248168] Call Trace: [ 8676.249653] [ 8676.256389] ? show_regs.cold.9+0x22/0x2f [ 8676.260235] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8676.264460] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8676.266887] ? watchdog+0x40/0x40 [ 8676.271320] ? hrtimer_interrupt+0x128/0x2c0 [ 8676.273669] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8676.276500] ? apic_timer_interrupt+0xf/0x20 [ 8676.286155] [ 8676.288238] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8676.292701] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8676.295948] do_raw_spin_lock+0x177/0x190 [ 8676.302071] _raw_spin_lock+0x21/0x30 [ 8676.305038] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8676.308332] ? do_raw_spin_unlock+0x75/0x190 [ 8676.312541] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8676.317565] ? do_wait_intr+0xf0/0xf0 [ 8676.320841] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8676.325325] mdc_reint+0x2d/0x110 [mdc] [ 8676.328521] mdc_unlink+0x658/0x870 [mdc] [ 8676.330430] lmv_unlink+0x34d/0xbd0 [lmv] [ 8676.337536] ll_unlink+0x24d/0x7f0 [lustre] [ 8676.347363] vfs_unlink+0x202/0x330 [ 8676.351463] do_unlinkat+0x2fa/0x460 [ 8676.355740] __x64_sys_unlink+0x37/0x50 [ 8676.359027] do_syscall_64+0xc1/0x440 [ 8676.361088] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8676.364407] RIP: 0033:0x7f83e5f5c10b [ 8676.370315] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8676.386888] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8676.397897] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8676.401675] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8676.412731] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8676.419187] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8676.424618] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8688.059030] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 8688.062478] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8688.088278] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8688.094622] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8688.098200] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8688.100517] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8688.110128] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8688.115138] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8688.119103] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8688.123473] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8688.127247] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8688.131217] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8688.135221] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8688.139567] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8688.142599] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8688.147531] Call Trace: [ 8688.149260] [ 8688.150360] ? show_regs.cold.9+0x22/0x2f [ 8688.152486] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8688.154794] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8688.157218] ? watchdog+0x40/0x40 [ 8688.158628] ? hrtimer_interrupt+0x128/0x2c0 [ 8688.161291] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8688.164337] ? apic_timer_interrupt+0xf/0x20 [ 8688.168263] [ 8688.171288] ? native_safe_halt+0x2e/0x50 [ 8688.178603] kvm_wait+0xcd/0xe0 [ 8688.183759] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8688.194166] do_raw_spin_lock+0x177/0x190 [ 8688.195719] _raw_spin_lock+0x21/0x30 [ 8688.199225] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8688.205654] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8688.209134] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8688.212707] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8688.216483] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8688.220026] ? woken_wake_function+0x30/0x30 [ 8688.222171] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8688.226083] kthread+0x1d1/0x200 [ 8688.227607] ? set_kthread_struct+0x70/0x70 [ 8688.229708] ret_from_fork+0x1f/0x30 [ 8692.056008] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 8692.059347] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8692.064385] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 8692.094225] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8692.094240] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8692.094243] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8692.094257] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8692.094262] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8692.094268] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8692.094271] RDX: 0000000000002e2b RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8692.094274] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8692.094276] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8692.094279] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8692.094283] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8692.094287] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8692.094290] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8692.094300] Call Trace: [ 8692.094305] [ 8692.094310] ? show_regs.cold.9+0x22/0x2f [ 8692.094320] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8692.094327] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8692.094332] ? watchdog+0x40/0x40 [ 8692.094336] ? hrtimer_interrupt+0x128/0x2c0 [ 8692.094341] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8692.094350] ? apic_timer_interrupt+0xf/0x20 [ 8692.094355] [ 8692.094358] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8692.094364] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8692.094369] do_raw_spin_lock+0x177/0x190 [ 8692.094374] _raw_spin_lock+0x21/0x30 [ 8692.094382] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8692.094682] obd_mod_free+0x46/0x140 [ptlrpc] [ 8692.095031] mdc_commit_open+0x91/0xb0 [mdc] [ 8692.095075] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8692.095290] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8692.095568] after_reply+0xe2c/0x1780 [ptlrpc] [ 8692.095810] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8692.096170] ? __next_timer_interrupt+0x160/0x160 [ 8692.096180] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8692.096410] ? do_wait_intr+0xf0/0xf0 [ 8692.096419] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8692.097995] kthread+0x1d1/0x200 [ 8692.098004] ? set_kthread_struct+0x70/0x70 [ 8692.098009] ret_from_fork+0x1f/0x30 [ 8692.303841] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8692.343973] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8692.349528] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8692.353068] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8692.356091] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8692.358383] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8692.365715] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8692.375294] RAX: 0000000000007e94 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8692.382293] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8692.391465] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8692.394465] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8692.397706] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8692.400132] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8692.402460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8692.411194] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8692.414936] Call Trace: [ 8692.416450] [ 8692.417575] ? show_regs.cold.9+0x22/0x2f [ 8692.419953] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8692.423286] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8692.429267] ? watchdog+0x40/0x40 [ 8692.434912] ? hrtimer_interrupt+0x128/0x2c0 [ 8692.439524] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8692.446216] ? apic_timer_interrupt+0xf/0x20 [ 8692.450499] [ 8692.452606] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8692.457512] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8692.460334] do_raw_spin_lock+0x177/0x190 [ 8692.462563] _raw_spin_lock+0x21/0x30 [ 8692.464141] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8692.470694] process_one_work+0x2c8/0x700 [ 8692.473613] worker_thread+0x296/0x6e0 [ 8692.478784] ? rescuer_thread+0x570/0x570 [ 8692.481393] kthread+0x1d1/0x200 [ 8692.482838] ? set_kthread_struct+0x70/0x70 [ 8692.484955] ret_from_fork+0x1f/0x30 [ 8695.777050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6182s! [ 8695.779933] Showing busy workqueues and worker pools: [ 8695.781478] workqueue events: flags=0x0 [ 8695.783184] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8695.785587] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8695.789773] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8695.792512] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8695.794943] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8695.797400] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8695.800115] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8695.802835] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8695.806796] workqueue events_long: flags=0x0 [ 8695.808776] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8695.810865] pending: lu_objects_flush [obdclass] [ 8695.812866] workqueue events_unbound: flags=0x2 [ 8695.814609] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8695.816859] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8695.819015] workqueue events_power_efficient: flags=0x80 [ 8695.820791] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8695.823426] pending: neigh_periodic_work, do_cache_clean [ 8695.825688] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8695.827739] pending: neigh_periodic_work, check_lifetime [ 8695.829622] workqueue mm_percpu_wq: flags=0x8 [ 8695.831026] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8695.833230] pending: vmstat_update, lru_add_drain_per_cpu [ 8695.835707] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8695.837926] pending: vmstat_update [ 8695.844056] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8695.848989] pending: lru_add_drain_per_cpu BAR(48) [ 8695.852277] workqueue ipv6_addrconf: flags=0x40008 [ 8695.854267] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8695.862265] pending: addrconf_verify_work [ 8695.863709] workqueue knbd0-recv: flags=0x1a [ 8695.867415] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8695.870086] in-flight: 650:recv_work [ 8695.871336] workqueue ptlrpc_pinger: flags=0x2 [ 8695.873169] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8695.875710] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8695.878181] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6182s workers=4 idle: 36 466 29757 [ 8695.881009] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8685s workers=2 idle: 652 [ 8701.120368] rcu: INFO: rcu_sched self-detected stall on CPU [ 8701.125101] rcu: 0-....: (4273159 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 8701.131812] (t=6720172 jiffies g=262441 q=59206) [ 8701.136620] rcu: rcu_sched kthread starved for 6189364 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 8701.141324] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8701.145740] rcu: RCU grace-period kthread stack dump: [ 8701.148744] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8701.159068] Call Trace: [ 8701.160199] __schedule+0x351/0xcb0 [ 8701.162429] schedule+0xc0/0x180 [ 8701.164373] schedule_timeout+0xb4/0x190 [ 8701.166077] ? do_raw_spin_unlock+0x75/0x190 [ 8701.169350] ? __next_timer_interrupt+0x160/0x160 [ 8701.172727] ? prepare_to_swait_event+0xf3/0x140 [ 8701.175204] rcu_gp_kthread+0x8dd/0x15b0 [ 8701.177637] ? force_qs_rnp+0x2c0/0x2c0 [ 8701.179929] kthread+0x1d1/0x200 [ 8701.181468] ? set_kthread_struct+0x70/0x70 [ 8701.188177] ret_from_fork+0x1f/0x30 [ 8701.192629] rcu: Stack dump where RCU GP kthread last ran: [ 8701.198008] Sending NMI from CPU 0 to CPUs 2: [ 8701.202330] NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0x2e/0x50 [ 8701.205728] NMI backtrace for cpu 0 [ 8701.207581] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8701.214301] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8701.222615] Call Trace: [ 8701.226133] [ 8701.228528] ? dump_stack+0xbb/0x10e [ 8701.232340] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8701.239854] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8701.245539] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8701.252532] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8701.255177] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8701.261625] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8701.263962] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8701.268644] ? update_process_times+0x6d/0xc0 [ 8701.270327] ? tick_sched_handle.isra.18+0x27/0x90 [ 8701.277335] ? tick_sched_timer+0x53/0xb0 [ 8701.281510] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8701.283373] ? tick_sched_handle.isra.18+0x90/0x90 [ 8701.285737] ? hrtimer_interrupt+0x128/0x2c0 [ 8701.287819] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8701.289905] ? apic_timer_interrupt+0xf/0x20 [ 8701.292494] [ 8701.293515] ? native_safe_halt+0x2e/0x50 [ 8701.297244] ? kvm_wait+0xcd/0xe0 [ 8701.300682] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8701.305910] ? do_raw_spin_lock+0x177/0x190 [ 8701.310685] ? _raw_spin_lock+0x21/0x30 [ 8701.314973] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8701.320305] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8701.325193] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8701.332544] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8701.339685] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8701.345420] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8701.351261] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8701.357632] ? __next_timer_interrupt+0x160/0x160 [ 8701.359917] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8701.365061] ? do_wait_intr+0xf0/0xf0 [ 8701.368689] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8701.372757] ? kthread+0x1d1/0x200 [ 8701.374211] ? set_kthread_struct+0x70/0x70 [ 8701.376392] ? ret_from_fork+0x1f/0x30 [ 8704.062062] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 8704.065425] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8704.103652] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8704.116276] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8704.125302] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8704.128629] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8704.143549] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8704.147472] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8704.150290] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8704.153405] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8704.156438] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8704.161717] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8704.166547] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8704.171136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8704.175210] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8704.178115] Call Trace: [ 8704.178707] [ 8704.179238] ? show_regs.cold.9+0x22/0x2f [ 8704.180885] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8704.182466] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8704.187198] ? watchdog+0x40/0x40 [ 8704.190534] ? hrtimer_interrupt+0x128/0x2c0 [ 8704.195100] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8704.197106] ? apic_timer_interrupt+0xf/0x20 [ 8704.202073] [ 8704.202838] ? native_safe_halt+0x2e/0x50 [ 8704.205415] kvm_wait+0xcd/0xe0 [ 8704.206502] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8704.211268] do_raw_spin_lock+0x177/0x190 [ 8704.214682] _raw_spin_lock+0x21/0x30 [ 8704.219370] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8704.221848] ? do_raw_spin_unlock+0x75/0x190 [ 8704.226915] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8704.231312] ? do_wait_intr+0xf0/0xf0 [ 8704.235132] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8704.239688] mdc_reint+0x2d/0x110 [mdc] [ 8704.243231] mdc_unlink+0x658/0x870 [mdc] [ 8704.245689] lmv_unlink+0x34d/0xbd0 [lmv] [ 8704.248443] ll_unlink+0x24d/0x7f0 [lustre] [ 8704.254703] vfs_unlink+0x202/0x330 [ 8704.262272] do_unlinkat+0x2fa/0x460 [ 8704.263613] __x64_sys_unlink+0x37/0x50 [ 8704.270996] do_syscall_64+0xc1/0x440 [ 8704.273334] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8704.279516] RIP: 0033:0x7f83e5f5c10b [ 8704.289687] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8704.298413] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8704.302391] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8704.308262] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8704.314660] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8704.318394] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8704.324397] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8716.059063] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 8716.062011] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8716.081296] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8716.085998] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8716.089494] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8716.092094] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8716.100248] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8716.106055] RAX: 0000000000005e58 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8716.111020] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8716.117107] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8716.122422] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8716.128355] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8716.136128] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8716.141785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8716.145112] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8716.150081] Call Trace: [ 8716.151985] [ 8716.153014] ? show_regs.cold.9+0x22/0x2f [ 8716.155414] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8716.158974] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8716.162044] ? watchdog+0x40/0x40 [ 8716.164191] ? hrtimer_interrupt+0x128/0x2c0 [ 8716.167645] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8716.171054] ? apic_timer_interrupt+0xf/0x20 [ 8716.174123] [ 8716.175620] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8716.180717] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8716.183797] do_raw_spin_lock+0x177/0x190 [ 8716.186395] _raw_spin_lock+0x21/0x30 [ 8716.188349] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8716.192275] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8716.194891] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8716.197858] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8716.200526] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8716.203440] ? woken_wake_function+0x30/0x30 [ 8716.205776] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8716.209680] kthread+0x1d1/0x200 [ 8716.211114] ? set_kthread_struct+0x70/0x70 [ 8716.212473] ret_from_fork+0x1f/0x30 [ 8725.988350] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6212s! [ 8726.000795] Showing busy workqueues and worker pools: [ 8726.007620] workqueue events: flags=0x0 [ 8726.011160] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8726.016020] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8726.020945] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8726.024905] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8726.028445] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8726.031947] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8726.036089] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8726.039445] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8726.044416] workqueue events_long: flags=0x0 [ 8726.046719] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8726.050163] pending: lu_objects_flush [obdclass] [ 8726.053304] workqueue events_unbound: flags=0x2 [ 8726.056351] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8726.059438] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8726.063293] workqueue events_power_efficient: flags=0x80 [ 8726.066486] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8726.070232] pending: neigh_periodic_work, do_cache_clean [ 8726.073205] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8726.077232] pending: neigh_periodic_work, check_lifetime [ 8726.080385] workqueue mm_percpu_wq: flags=0x8 [ 8726.082922] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8726.086861] pending: vmstat_update, lru_add_drain_per_cpu [ 8726.090269] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8726.096184] pending: vmstat_update [ 8726.099715] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8726.106819] pending: lru_add_drain_per_cpu BAR(48) [ 8726.111601] workqueue ipv6_addrconf: flags=0x40008 [ 8726.114286] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8726.118755] pending: addrconf_verify_work [ 8726.121304] workqueue knbd0-recv: flags=0x1a [ 8726.124977] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8726.128781] in-flight: 650:recv_work [ 8726.130429] workqueue ptlrpc_pinger: flags=0x2 [ 8726.132724] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8726.135946] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8726.138910] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6212s workers=4 idle: 36 466 29757 [ 8726.142467] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8716s workers=2 idle: 652 [ 8728.063643] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 8728.066992] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8728.124978] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8728.141012] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8728.149898] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8728.158015] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8728.161682] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8728.174475] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8728.186175] RAX: 00000000000000a2 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8728.192455] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8728.202132] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8728.209885] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8728.214694] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8728.219022] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8728.230907] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8728.237787] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8728.243721] Call Trace: [ 8728.244673] [ 8728.245430] ? show_regs.cold.9+0x22/0x2f [ 8728.253245] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8728.258756] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8728.267364] ? watchdog+0x40/0x40 [ 8728.268639] ? hrtimer_interrupt+0x128/0x2c0 [ 8728.273466] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8728.275971] ? apic_timer_interrupt+0xf/0x20 [ 8728.277942] [ 8728.279780] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8728.283058] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8728.285825] do_raw_spin_lock+0x177/0x190 [ 8728.292559] _raw_spin_lock+0x21/0x30 [ 8728.301817] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8728.304005] process_one_work+0x2c8/0x700 [ 8728.311443] worker_thread+0x296/0x6e0 [ 8728.320674] ? rescuer_thread+0x570/0x570 [ 8728.323264] kthread+0x1d1/0x200 [ 8728.326104] ? set_kthread_struct+0x70/0x70 [ 8728.328190] ret_from_fork+0x1f/0x30 [ 8732.062287] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8732.075309] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8732.110046] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8732.116037] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8732.119106] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8732.121416] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8732.127805] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 8732.136202] RAX: 0000000000007838 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8732.138698] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8732.143194] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8732.151036] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8732.161943] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8732.177032] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8732.189805] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8732.202410] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8732.204603] Call Trace: [ 8732.210165] [ 8732.211581] ? show_regs.cold.9+0x22/0x2f [ 8732.212837] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8732.217662] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8732.222488] ? watchdog+0x40/0x40 [ 8732.226346] ? hrtimer_interrupt+0x128/0x2c0 [ 8732.230464] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8732.233791] ? apic_timer_interrupt+0xf/0x20 [ 8732.238013] [ 8732.238989] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8732.244531] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8732.246882] do_raw_spin_lock+0x177/0x190 [ 8732.248343] _raw_spin_lock+0x21/0x30 [ 8732.250925] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8732.255968] ? do_raw_spin_unlock+0x75/0x190 [ 8732.258564] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8732.261140] ? do_wait_intr+0xf0/0xf0 [ 8732.265984] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8732.270113] mdc_reint+0x2d/0x110 [mdc] [ 8732.272650] mdc_unlink+0x658/0x870 [mdc] [ 8732.276321] lmv_unlink+0x34d/0xbd0 [lmv] [ 8732.278656] ll_unlink+0x24d/0x7f0 [lustre] [ 8732.282948] vfs_unlink+0x202/0x330 [ 8732.286878] do_unlinkat+0x2fa/0x460 [ 8732.291059] __x64_sys_unlink+0x37/0x50 [ 8732.295124] do_syscall_64+0xc1/0x440 [ 8732.300970] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8732.309138] RIP: 0033:0x7f83e5f5c10b [ 8732.311055] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8732.330735] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8732.337351] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8732.346068] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8732.353756] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8732.363514] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8732.371462] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8744.060166] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 8744.069266] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8744.138581] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8744.164357] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8744.171976] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8744.182205] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8744.212184] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8744.226814] RAX: 0000000000004844 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8744.245483] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8744.254693] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8744.262469] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8744.277463] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8744.290039] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8744.297538] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8744.312625] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8744.322260] Call Trace: [ 8744.323757] [ 8744.325292] ? show_regs.cold.9+0x22/0x2f [ 8744.331979] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8744.338486] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8744.344400] ? watchdog+0x40/0x40 [ 8744.346115] ? hrtimer_interrupt+0x128/0x2c0 [ 8744.356192] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8744.369727] ? apic_timer_interrupt+0xf/0x20 [ 8744.373286] [ 8744.379631] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8744.392138] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8744.394555] do_raw_spin_lock+0x177/0x190 [ 8744.402715] _raw_spin_lock+0x21/0x30 [ 8744.406597] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8744.414758] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8744.420664] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8744.427762] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8744.438028] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8744.448858] ? woken_wake_function+0x30/0x30 [ 8744.456388] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8744.463877] kthread+0x1d1/0x200 [ 8744.467691] ? set_kthread_struct+0x70/0x70 [ 8744.472094] ret_from_fork+0x1f/0x30 [ 8752.056030] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 8752.058885] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8752.079909] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8752.086137] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8752.091179] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8752.093766] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8752.100494] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8752.103847] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8752.106972] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8752.120375] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8752.130463] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8752.141734] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8752.154678] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8752.161971] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8752.166272] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8752.171673] Call Trace: [ 8752.175204] [ 8752.178449] ? show_regs.cold.9+0x22/0x2f [ 8752.187210] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8752.188970] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8752.202258] ? watchdog+0x40/0x40 [ 8752.203507] ? hrtimer_interrupt+0x128/0x2c0 [ 8752.204854] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8752.213744] ? apic_timer_interrupt+0xf/0x20 [ 8752.216630] [ 8752.220689] ? native_safe_halt+0x2e/0x50 [ 8752.230363] kvm_wait+0xcd/0xe0 [ 8752.231462] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8752.235354] do_raw_spin_lock+0x177/0x190 [ 8752.236934] _raw_spin_lock+0x21/0x30 [ 8752.238440] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8752.241952] obd_mod_free+0x46/0x140 [ptlrpc] [ 8752.251591] mdc_commit_open+0x91/0xb0 [mdc] [ 8752.256714] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8752.265875] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8752.271808] after_reply+0xe2c/0x1780 [ptlrpc] [ 8752.277460] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8752.283530] ? __next_timer_interrupt+0x160/0x160 [ 8752.288619] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8752.296609] ? do_wait_intr+0xf0/0xf0 [ 8752.298904] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8752.306691] kthread+0x1d1/0x200 [ 8752.311976] ? set_kthread_struct+0x70/0x70 [ 8752.318715] ret_from_fork+0x1f/0x30 [ 8756.060369] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 8756.073071] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8756.168335] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8756.180339] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8756.189782] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8756.192055] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6243s! [ 8756.192077] Showing busy workqueues and worker pools: [ 8756.192079] workqueue events: flags=0x0 [ 8756.192083] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8756.192118] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8756.192508] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8756.192518] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8756.192649] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8756.192658] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8756.193073] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8756.193083] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8756.193312] workqueue events_long: flags=0x0 [ 8756.193315] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8756.193323] pending: lu_objects_flush [obdclass] [ 8756.199734] [ 8756.199742] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8756.199759] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8756.199763] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8756.199767] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8756.199770] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8756.199772] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8756.199774] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8756.199777] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8756.199780] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8756.199782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8756.199784] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8756.199792] Call Trace: [ 8756.199797] [ 8756.199802] ? show_regs.cold.9+0x22/0x2f [ 8756.199812] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8756.199818] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8756.199822] ? watchdog+0x40/0x40 [ 8756.199826] ? hrtimer_interrupt+0x128/0x2c0 [ 8756.199830] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8756.199837] ? apic_timer_interrupt+0xf/0x20 [ 8756.199842] [ 8756.199845] ? native_safe_halt+0x2e/0x50 [ 8756.199849] kvm_wait+0xcd/0xe0 [ 8756.199856] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8756.199865] do_raw_spin_lock+0x177/0x190 [ 8756.199870] _raw_spin_lock+0x21/0x30 [ 8756.199875] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8756.204771] [ 8756.204807] workqueue events_unbound: flags=0x2 [ 8756.204815] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8756.204828] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8756.205381] workqueue events_power_efficient: flags=0x80 [ 8756.205385] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8756.205394] pending: neigh_periodic_work, do_cache_clean [ 8756.205409] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8756.205417] pending: neigh_periodic_work, check_lifetime [ 8756.205429] workqueue mm_percpu_wq: flags=0x8 [ 8756.205432] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8756.205463] pending: vmstat_update, lru_add_drain_per_cpu [ 8756.205474] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8756.205724] pending: vmstat_update [ 8756.205731] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8756.205738] pending: lru_add_drain_per_cpu BAR(48) [ 8756.205764] workqueue ipv6_addrconf: flags=0x40008 [ 8756.205767] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8756.205774] pending: addrconf_verify_work [ 8756.205787] workqueue knbd0-recv: flags=0x1a [ 8756.205791] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8756.205798] in-flight: 650:recv_work [ 8756.205808] workqueue ptlrpc_pinger: flags=0x2 [ 8756.205811] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8756.205826] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8756.206229] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6243s workers=4 idle: 36 466 29757 [ 8756.206248] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8746s workers=2 idle: 652 [ 8756.864279] process_one_work+0x2c8/0x700 [ 8756.866806] worker_thread+0x296/0x6e0 [ 8756.868836] ? rescuer_thread+0x570/0x570 [ 8756.873966] kthread+0x1d1/0x200 [ 8756.878425] ? set_kthread_struct+0x70/0x70 [ 8756.883480] ret_from_fork+0x1f/0x30 [ 8760.072344] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8760.139260] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8760.228035] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8760.267706] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8760.283140] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8760.317257] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8760.347862] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8760.364817] RAX: 000000000000710b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8760.384361] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8760.397787] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8760.403236] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8760.413368] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8760.422636] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8760.441797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8760.457448] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8760.468933] Call Trace: [ 8760.470235] [ 8760.471393] ? show_regs.cold.9+0x22/0x2f [ 8760.479031] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8760.505338] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8760.509601] ? watchdog+0x40/0x40 [ 8760.514761] ? hrtimer_interrupt+0x128/0x2c0 [ 8760.521624] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8760.527880] ? apic_timer_interrupt+0xf/0x20 [ 8760.531273] [ 8760.544070] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8760.545827] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8760.549403] do_raw_spin_lock+0x177/0x190 [ 8760.557117] _raw_spin_lock+0x21/0x30 [ 8760.559934] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8760.561908] ? do_raw_spin_unlock+0x75/0x190 [ 8760.572879] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8760.582493] ? do_wait_intr+0xf0/0xf0 [ 8760.585098] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8760.594814] mdc_reint+0x2d/0x110 [mdc] [ 8760.605659] mdc_unlink+0x658/0x870 [mdc] [ 8760.607708] lmv_unlink+0x34d/0xbd0 [lmv] [ 8760.616150] ll_unlink+0x24d/0x7f0 [lustre] [ 8760.617467] vfs_unlink+0x202/0x330 [ 8760.618440] do_unlinkat+0x2fa/0x460 [ 8760.626689] __x64_sys_unlink+0x37/0x50 [ 8760.630754] do_syscall_64+0xc1/0x440 [ 8760.637118] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8760.638490] RIP: 0033:0x7f83e5f5c10b [ 8760.646773] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8760.684358] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8760.730609] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8760.732553] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8760.739072] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8760.742113] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8760.745749] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8772.060674] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 8772.080602] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8772.141304] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8772.160611] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8772.172494] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8772.182480] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8772.214477] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8772.225205] RAX: 0000000000007a07 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8772.234324] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8772.236381] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8772.247437] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8772.258449] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8772.273466] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8772.278283] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8772.282454] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8772.290968] Call Trace: [ 8772.296206] [ 8772.297639] ? show_regs.cold.9+0x22/0x2f [ 8772.313498] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8772.316261] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8772.327703] ? watchdog+0x40/0x40 [ 8772.328803] ? hrtimer_interrupt+0x128/0x2c0 [ 8772.330202] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8772.340560] ? apic_timer_interrupt+0xf/0x20 [ 8772.342509] [ 8772.352912] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8772.356040] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8772.361590] do_raw_spin_lock+0x177/0x190 [ 8772.363520] _raw_spin_lock+0x21/0x30 [ 8772.369457] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8772.381094] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8772.390018] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8772.401461] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8772.404414] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8772.407728] ? woken_wake_function+0x30/0x30 [ 8772.417592] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8772.421766] kthread+0x1d1/0x200 [ 8772.423470] ? set_kthread_struct+0x70/0x70 [ 8772.426423] ret_from_fork+0x1f/0x30 [ 8784.064621] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 8784.067853] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 8784.073357] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8784.073440] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8784.073448] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8784.081316] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8784.081395] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8784.081401] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8784.081407] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8784.108202] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8784.108228] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8784.108232] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8784.108238] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8784.108240] RDX: 00000000000054dc RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8784.108243] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8784.108245] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8784.108248] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8784.108251] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8784.108254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8784.108256] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8784.108263] Call Trace: [ 8784.108269] [ 8784.108275] ? show_regs.cold.9+0x22/0x2f [ 8784.108284] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8784.108291] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8784.108295] ? watchdog+0x40/0x40 [ 8784.108299] ? hrtimer_interrupt+0x128/0x2c0 [ 8784.108304] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8784.108313] ? apic_timer_interrupt+0xf/0x20 [ 8784.108317] [ 8784.108321] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8784.108326] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8784.108331] do_raw_spin_lock+0x177/0x190 [ 8784.108335] _raw_spin_lock+0x21/0x30 [ 8784.108342] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8784.108664] obd_mod_free+0x46/0x140 [ptlrpc] [ 8784.108928] mdc_commit_open+0x91/0xb0 [mdc] [ 8784.109262] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8784.109632] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8784.133147] [ 8784.133158] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8784.133177] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8784.133181] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8784.133186] RAX: 000000000000680b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8784.133189] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8784.133191] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8784.133193] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8784.133195] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8784.133198] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8784.133201] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8784.133204] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8784.133217] Call Trace: [ 8784.133223] [ 8784.133228] ? show_regs.cold.9+0x22/0x2f [ 8784.133237] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8784.133244] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8784.133248] ? watchdog+0x40/0x40 [ 8784.133252] ? hrtimer_interrupt+0x128/0x2c0 [ 8784.133256] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8784.133264] ? apic_timer_interrupt+0xf/0x20 [ 8784.133269] [ 8784.133272] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8784.133278] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8784.133284] do_raw_spin_lock+0x177/0x190 [ 8784.133295] _raw_spin_lock+0x21/0x30 [ 8784.133299] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8784.134104] process_one_work+0x2c8/0x700 [ 8784.134112] worker_thread+0x296/0x6e0 [ 8784.134116] ? rescuer_thread+0x570/0x570 [ 8784.134120] kthread+0x1d1/0x200 [ 8784.134125] ? set_kthread_struct+0x70/0x70 [ 8784.134130] ret_from_fork+0x1f/0x30 [ 8784.427299] after_reply+0xe2c/0x1780 [ptlrpc] [ 8784.431063] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8784.434114] ? __next_timer_interrupt+0x160/0x160 [ 8784.436997] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8784.439656] ? do_wait_intr+0xf0/0xf0 [ 8784.441606] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8784.444614] kthread+0x1d1/0x200 [ 8784.448258] ? set_kthread_struct+0x70/0x70 [ 8784.450273] ret_from_fork+0x1f/0x30 [ 8786.407818] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6273s! [ 8786.418948] Showing busy workqueues and worker pools: [ 8786.424081] workqueue events: flags=0x0 [ 8786.429467] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8786.437180] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8786.445280] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8786.449765] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8786.453050] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8786.456172] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8786.460014] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8786.466097] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8786.470601] workqueue events_long: flags=0x0 [ 8786.473197] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8786.476147] pending: lu_objects_flush [obdclass] [ 8786.478692] workqueue events_unbound: flags=0x2 [ 8786.482558] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8786.489542] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8786.501362] workqueue events_power_efficient: flags=0x80 [ 8786.502901] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8786.514821] pending: neigh_periodic_work, do_cache_clean [ 8786.518153] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8786.521212] pending: neigh_periodic_work, check_lifetime [ 8786.526807] workqueue mm_percpu_wq: flags=0x8 [ 8786.533106] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8786.536843] pending: vmstat_update, lru_add_drain_per_cpu [ 8786.543285] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8786.551380] pending: vmstat_update [ 8786.555605] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8786.567448] pending: lru_add_drain_per_cpu BAR(48) [ 8786.573094] workqueue ipv6_addrconf: flags=0x40008 [ 8786.581903] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8786.590282] pending: addrconf_verify_work [ 8786.593523] workqueue knbd0-recv: flags=0x1a [ 8786.595858] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8786.598858] in-flight: 650:recv_work [ 8786.605845] workqueue ptlrpc_pinger: flags=0x2 [ 8786.614032] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8786.616605] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8786.620705] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6273s workers=4 idle: 36 466 29757 [ 8786.624251] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8776s workers=2 idle: 652 [ 8788.064494] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8788.079851] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8788.125354] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8788.136032] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8788.140446] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8788.144978] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8788.161662] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8788.174035] RAX: 000000000000281c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8788.182330] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8788.190794] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8788.193545] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8788.197103] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8788.201027] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8788.204127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8788.208381] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8788.212890] Call Trace: [ 8788.213840] [ 8788.218852] ? show_regs.cold.9+0x22/0x2f [ 8788.221267] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8788.227389] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8788.231615] ? watchdog+0x40/0x40 [ 8788.233467] ? hrtimer_interrupt+0x128/0x2c0 [ 8788.236634] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8788.241331] ? apic_timer_interrupt+0xf/0x20 [ 8788.245595] [ 8788.247694] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8788.249619] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8788.255131] do_raw_spin_lock+0x177/0x190 [ 8788.258084] _raw_spin_lock+0x21/0x30 [ 8788.259526] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8788.264160] ? do_raw_spin_unlock+0x75/0x190 [ 8788.266690] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8788.273093] ? do_wait_intr+0xf0/0xf0 [ 8788.278366] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8788.281154] mdc_reint+0x2d/0x110 [mdc] [ 8788.282557] mdc_unlink+0x658/0x870 [mdc] [ 8788.284790] lmv_unlink+0x34d/0xbd0 [lmv] [ 8788.286344] ll_unlink+0x24d/0x7f0 [lustre] [ 8788.288125] vfs_unlink+0x202/0x330 [ 8788.289357] do_unlinkat+0x2fa/0x460 [ 8788.290661] __x64_sys_unlink+0x37/0x50 [ 8788.292169] do_syscall_64+0xc1/0x440 [ 8788.293783] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8788.297261] RIP: 0033:0x7f83e5f5c10b [ 8788.298719] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8788.316033] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8788.325530] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8788.330507] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8788.342990] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8788.346213] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8788.349087] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8800.059254] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 8800.062261] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8800.116928] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8800.128349] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8800.136888] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8800.142071] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8800.162979] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8800.173749] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8800.178882] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8800.191933] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8800.202859] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8800.217309] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8800.233414] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8800.242078] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8800.252929] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8800.255443] Call Trace: [ 8800.262104] [ 8800.264130] ? show_regs.cold.9+0x22/0x2f [ 8800.270519] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8800.275045] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8800.278253] ? watchdog+0x40/0x40 [ 8800.280176] ? hrtimer_interrupt+0x128/0x2c0 [ 8800.284327] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8800.285937] ? apic_timer_interrupt+0xf/0x20 [ 8800.291962] [ 8800.292923] ? native_safe_halt+0x2e/0x50 [ 8800.296651] kvm_wait+0xcd/0xe0 [ 8800.298252] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8800.303214] do_raw_spin_lock+0x177/0x190 [ 8800.306207] _raw_spin_lock+0x21/0x30 [ 8800.310179] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8800.314983] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8800.319352] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8800.326777] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8800.332114] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8800.333928] ? woken_wake_function+0x30/0x30 [ 8800.335386] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8800.345607] kthread+0x1d1/0x200 [ 8800.347941] ? set_kthread_struct+0x70/0x70 [ 8800.349432] ret_from_fork+0x1f/0x30 [ 8812.056060] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 8812.060057] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 8812.064401] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8812.064489] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8812.064496] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8812.064499] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8812.064513] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8812.064517] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8812.064521] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8812.064524] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8812.064526] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8812.064528] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8812.064530] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8812.064533] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8812.064536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8812.064538] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8812.064546] Call Trace: [ 8812.064552] [ 8812.064557] ? show_regs.cold.9+0x22/0x2f [ 8812.064566] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8812.064573] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8812.064577] ? watchdog+0x40/0x40 [ 8812.064582] ? hrtimer_interrupt+0x128/0x2c0 [ 8812.064585] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8812.064591] ? apic_timer_interrupt+0xf/0x20 [ 8812.064595] [ 8812.064598] ? native_safe_halt+0x2e/0x50 [ 8812.064602] kvm_wait+0xcd/0xe0 [ 8812.064609] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8812.064617] do_raw_spin_lock+0x177/0x190 [ 8812.064620] _raw_spin_lock+0x21/0x30 [ 8812.064625] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8812.065190] obd_mod_free+0x46/0x140 [ptlrpc] [ 8812.065389] mdc_commit_open+0x91/0xb0 [mdc] [ 8812.065425] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8812.065698] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8812.066151] after_reply+0xe2c/0x1780 [ptlrpc] [ 8812.066350] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8812.066694] ? __next_timer_interrupt+0x160/0x160 [ 8812.066704] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8812.067038] ? do_wait_intr+0xf0/0xf0 [ 8812.067048] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8812.067244] kthread+0x1d1/0x200 [ 8812.067250] ? set_kthread_struct+0x70/0x70 [ 8812.067253] ret_from_fork+0x1f/0x30 [ 8812.259944] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8812.280963] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8812.288784] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8812.293638] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8812.298453] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8812.300276] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8812.311392] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8812.313383] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8812.318561] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8812.321518] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8812.323902] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8812.333990] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8812.346043] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8812.352068] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8812.360856] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8812.370144] Call Trace: [ 8812.372269] [ 8812.372887] ? show_regs.cold.9+0x22/0x2f [ 8812.374097] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8812.375348] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8812.381926] ? watchdog+0x40/0x40 [ 8812.383449] ? hrtimer_interrupt+0x128/0x2c0 [ 8812.384982] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8812.386568] ? apic_timer_interrupt+0xf/0x20 [ 8812.388167] [ 8812.389046] ? native_safe_halt+0x2e/0x50 [ 8812.390771] kvm_wait+0xcd/0xe0 [ 8812.394559] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8812.399051] do_raw_spin_lock+0x177/0x190 [ 8812.400811] _raw_spin_lock+0x21/0x30 [ 8812.405621] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8812.410168] process_one_work+0x2c8/0x700 [ 8812.412630] worker_thread+0x296/0x6e0 [ 8812.415036] ? rescuer_thread+0x570/0x570 [ 8812.416648] kthread+0x1d1/0x200 [ 8812.420665] ? set_kthread_struct+0x70/0x70 [ 8812.422460] ret_from_fork+0x1f/0x30 [ 8816.062029] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8816.068520] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8816.136211] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8816.167979] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8816.175929] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8816.178945] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8816.192018] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8816.203518] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8816.209462] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8816.225294] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8816.242258] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8816.261907] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8816.279984] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8816.286880] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8816.299931] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8816.302379] Call Trace: [ 8816.303295] [ 8816.309285] ? show_regs.cold.9+0x22/0x2f [ 8816.316726] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8816.320054] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8816.325305] ? watchdog+0x40/0x40 [ 8816.328879] ? hrtimer_interrupt+0x128/0x2c0 [ 8816.330382] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8816.333956] ? apic_timer_interrupt+0xf/0x20 [ 8816.354158] [ 8816.359168] ? native_safe_halt+0x2e/0x50 [ 8816.368757] kvm_wait+0xcd/0xe0 [ 8816.382630] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8816.384221] do_raw_spin_lock+0x177/0x190 [ 8816.402801] _raw_spin_lock+0x21/0x30 [ 8816.417128] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8816.426508] ? do_raw_spin_unlock+0x75/0x190 [ 8816.434976] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8816.439859] ? do_wait_intr+0xf0/0xf0 [ 8816.441196] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8816.457833] mdc_reint+0x2d/0x110 [mdc] [ 8816.463545] mdc_unlink+0x658/0x870 [mdc] [ 8816.465102] lmv_unlink+0x34d/0xbd0 [lmv] [ 8816.481385] ll_unlink+0x24d/0x7f0 [lustre] [ 8816.483049] vfs_unlink+0x202/0x330 [ 8816.506910] do_unlinkat+0x2fa/0x460 [ 8816.511296] __x64_sys_unlink+0x37/0x50 [ 8816.512993] do_syscall_64+0xc1/0x440 [ 8816.514948] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8816.518757] RIP: 0033:0x7f83e5f5c10b [ 8816.520071] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8816.542703] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8816.550608] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8816.555580] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8816.559352] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8816.565787] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8816.571085] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8817.123105] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6303s! [ 8817.130446] Showing busy workqueues and worker pools: [ 8817.134415] workqueue events: flags=0x0 [ 8817.135953] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8817.139092] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8817.143604] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8817.146961] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8817.150561] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8817.153420] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8817.157191] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8817.160215] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8817.164982] workqueue events_long: flags=0x0 [ 8817.167173] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8817.169946] pending: lu_objects_flush [obdclass] [ 8817.172800] workqueue events_unbound: flags=0x2 [ 8817.175066] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8817.177832] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8817.180527] workqueue events_power_efficient: flags=0x80 [ 8817.183059] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8817.185872] pending: neigh_periodic_work, do_cache_clean [ 8817.188513] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8817.191118] pending: neigh_periodic_work, check_lifetime [ 8817.193721] workqueue mm_percpu_wq: flags=0x8 [ 8817.195946] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8817.199095] pending: vmstat_update, lru_add_drain_per_cpu [ 8817.201531] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8817.205212] pending: vmstat_update [ 8817.206775] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8817.209280] pending: lru_add_drain_per_cpu BAR(48) [ 8817.211571] workqueue ipv6_addrconf: flags=0x40008 [ 8817.213673] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8817.216374] pending: addrconf_verify_work [ 8817.217855] workqueue knbd0-recv: flags=0x1a [ 8817.219552] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8817.222120] in-flight: 650:recv_work [ 8817.223920] workqueue ptlrpc_pinger: flags=0x2 [ 8817.225924] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8817.227796] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8817.230102] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6304s workers=4 idle: 36 466 29757 [ 8817.233623] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8807s workers=2 idle: 652 [ 8828.059169] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 8828.065333] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8828.097792] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8828.105025] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8828.110375] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8828.113567] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8828.122857] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8828.129680] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8828.135309] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8828.137084] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8828.144855] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8828.152029] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8828.156398] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8828.162827] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8828.169023] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8828.175741] Call Trace: [ 8828.176979] [ 8828.177685] ? show_regs.cold.9+0x22/0x2f [ 8828.181371] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8828.185251] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8828.190979] ? watchdog+0x40/0x40 [ 8828.192163] ? hrtimer_interrupt+0x128/0x2c0 [ 8828.194733] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8828.199590] ? apic_timer_interrupt+0xf/0x20 [ 8828.200711] [ 8828.201311] ? native_safe_halt+0x2e/0x50 [ 8828.204380] kvm_wait+0xcd/0xe0 [ 8828.206684] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8828.211011] do_raw_spin_lock+0x177/0x190 [ 8828.213904] _raw_spin_lock+0x21/0x30 [ 8828.217741] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8828.224006] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8828.228986] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8828.233919] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8828.237574] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8828.241950] ? woken_wake_function+0x30/0x30 [ 8828.245758] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8828.251840] kthread+0x1d1/0x200 [ 8828.254484] ? set_kthread_struct+0x70/0x70 [ 8828.255544] ret_from_fork+0x1f/0x30 [ 8840.060028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 8840.075333] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8840.117353] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8840.121255] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8840.125730] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8840.129866] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8840.131601] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8840.143013] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8840.145790] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8840.149235] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8840.153652] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8840.157822] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8840.162479] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8840.166299] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8840.175234] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8840.179686] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8840.185661] Call Trace: [ 8840.187233] [ 8840.190473] ? show_regs.cold.9+0x22/0x2f [ 8840.192768] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8840.195517] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8840.200397] ? watchdog+0x40/0x40 [ 8840.204154] ? hrtimer_interrupt+0x128/0x2c0 [ 8840.206938] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8840.208822] ? apic_timer_interrupt+0xf/0x20 [ 8840.210990] [ 8840.211964] ? native_safe_halt+0x2e/0x50 [ 8840.214811] kvm_wait+0xcd/0xe0 [ 8840.217875] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8840.222678] do_raw_spin_lock+0x177/0x190 [ 8840.227009] _raw_spin_lock+0x21/0x30 [ 8840.230486] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8840.235369] process_one_work+0x2c8/0x700 [ 8840.239544] worker_thread+0x296/0x6e0 [ 8840.242927] ? rescuer_thread+0x570/0x570 [ 8840.247951] kthread+0x1d1/0x200 [ 8840.254032] ? set_kthread_struct+0x70/0x70 [ 8840.258750] ret_from_fork+0x1f/0x30 [ 8844.058365] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 8844.065900] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8844.073566] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8844.092432] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8844.092445] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8844.092448] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8844.092463] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8844.092467] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8844.092472] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8844.092475] RDX: 0000000000007e13 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8844.092477] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8844.092480] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8844.092483] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8844.092486] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8844.092489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8844.092492] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8844.092500] Call Trace: [ 8844.092505] [ 8844.092511] ? show_regs.cold.9+0x22/0x2f [ 8844.092519] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8844.092526] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8844.092530] ? watchdog+0x40/0x40 [ 8844.092534] ? hrtimer_interrupt+0x128/0x2c0 [ 8844.092539] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8844.092547] ? apic_timer_interrupt+0xf/0x20 [ 8844.092551] [ 8844.092555] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8844.092560] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8844.092565] do_raw_spin_lock+0x177/0x190 [ 8844.092569] _raw_spin_lock+0x21/0x30 [ 8844.092576] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8844.092855] obd_mod_free+0x46/0x140 [ptlrpc] [ 8844.093265] mdc_commit_open+0x91/0xb0 [mdc] [ 8844.093307] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8844.093560] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8844.093830] after_reply+0xe2c/0x1780 [ptlrpc] [ 8844.094181] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8844.094447] ? __next_timer_interrupt+0x160/0x160 [ 8844.094457] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8844.094666] ? do_wait_intr+0xf0/0xf0 [ 8844.094676] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8844.094875] kthread+0x1d1/0x200 [ 8844.094880] ? set_kthread_struct+0x70/0x70 [ 8844.094884] ret_from_fork+0x1f/0x30 [ 8844.306103] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8844.350899] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8844.363458] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8844.368983] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8844.374873] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8844.394675] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8844.403619] RAX: 0000000000001518 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8844.411301] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8844.423678] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8844.434854] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8844.439352] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8844.446205] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8844.452794] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8844.460876] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8844.466120] Call Trace: [ 8844.467632] [ 8844.469701] ? show_regs.cold.9+0x22/0x2f [ 8844.471293] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8844.473366] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8844.477325] ? watchdog+0x40/0x40 [ 8844.479257] ? hrtimer_interrupt+0x128/0x2c0 [ 8844.482783] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8844.485048] ? apic_timer_interrupt+0xf/0x20 [ 8844.489319] [ 8844.490380] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8844.495119] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8844.497244] do_raw_spin_lock+0x177/0x190 [ 8844.500707] _raw_spin_lock+0x21/0x30 [ 8844.502218] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8844.507630] ? do_raw_spin_unlock+0x75/0x190 [ 8844.509989] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8844.512887] ? do_wait_intr+0xf0/0xf0 [ 8844.516724] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8844.520635] mdc_reint+0x2d/0x110 [mdc] [ 8844.522409] mdc_unlink+0x658/0x870 [mdc] [ 8844.528890] lmv_unlink+0x34d/0xbd0 [lmv] [ 8844.533729] ll_unlink+0x24d/0x7f0 [lustre] [ 8844.537789] vfs_unlink+0x202/0x330 [ 8844.539348] do_unlinkat+0x2fa/0x460 [ 8844.540894] __x64_sys_unlink+0x37/0x50 [ 8844.544816] do_syscall_64+0xc1/0x440 [ 8844.547830] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8844.553591] RIP: 0033:0x7f83e5f5c10b [ 8844.555996] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8844.575758] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8844.582950] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8844.594678] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8844.603775] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8844.610612] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8844.620866] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8847.338815] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6334s! [ 8847.349847] Showing busy workqueues and worker pools: [ 8847.362321] workqueue events: flags=0x0 [ 8847.369158] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8847.380953] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8847.396624] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8847.409723] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8847.427414] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8847.434311] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8847.454689] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8847.475584] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8847.485047] workqueue events_long: flags=0x0 [ 8847.500632] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8847.507752] pending: lu_objects_flush [obdclass] [ 8847.515520] workqueue events_unbound: flags=0x2 [ 8847.517859] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8847.520295] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8847.521833] workqueue events_power_efficient: flags=0x80 [ 8847.523541] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8847.526307] pending: neigh_periodic_work, do_cache_clean [ 8847.528667] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8847.531717] pending: neigh_periodic_work, check_lifetime [ 8847.533662] workqueue mm_percpu_wq: flags=0x8 [ 8847.535933] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8847.538376] pending: vmstat_update, lru_add_drain_per_cpu [ 8847.546344] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8847.551050] pending: vmstat_update [ 8847.556698] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8847.565223] pending: lru_add_drain_per_cpu BAR(48) [ 8847.573075] workqueue ipv6_addrconf: flags=0x40008 [ 8847.575606] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8847.579181] pending: addrconf_verify_work [ 8847.580810] workqueue knbd0-recv: flags=0x1a [ 8847.587276] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8847.593761] in-flight: 650:recv_work [ 8847.596461] workqueue ptlrpc_pinger: flags=0x2 [ 8847.597565] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8847.599028] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8847.606761] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6334s workers=4 idle: 36 466 29757 [ 8847.611494] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8837s workers=2 idle: 652 [ 8856.059063] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 8856.062793] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8856.108863] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8856.120271] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8856.132837] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8856.139746] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8856.159203] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8856.164200] RAX: 0000000000003cbf RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8856.171922] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8856.175074] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8856.179259] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8856.182993] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8856.187554] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8856.192551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8856.196298] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8856.201102] Call Trace: [ 8856.202254] [ 8856.203816] ? show_regs.cold.9+0x22/0x2f [ 8856.206210] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8856.208678] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8856.214972] ? watchdog+0x40/0x40 [ 8856.217843] ? hrtimer_interrupt+0x128/0x2c0 [ 8856.219950] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8856.223385] ? apic_timer_interrupt+0xf/0x20 [ 8856.225444] [ 8856.226702] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8856.229701] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8856.233230] do_raw_spin_lock+0x177/0x190 [ 8856.235808] _raw_spin_lock+0x21/0x30 [ 8856.239216] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8856.245979] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8856.251513] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8856.258546] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8856.262557] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8856.265535] ? woken_wake_function+0x30/0x30 [ 8856.266776] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8856.269368] kthread+0x1d1/0x200 [ 8856.270393] ? set_kthread_struct+0x70/0x70 [ 8856.271719] ret_from_fork+0x1f/0x30 [ 8868.060603] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 8868.065122] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8868.086527] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8868.091539] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8868.096612] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8868.100011] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8868.102295] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8868.111558] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8868.115129] RAX: 0000000000007af6 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8868.118613] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8868.122200] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8868.125658] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8868.129324] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8868.133404] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8868.137274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8868.139892] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8868.143519] Call Trace: [ 8868.144527] [ 8868.145318] ? show_regs.cold.9+0x22/0x2f [ 8868.147475] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8868.149423] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8868.152231] ? watchdog+0x40/0x40 [ 8868.153636] ? hrtimer_interrupt+0x128/0x2c0 [ 8868.156301] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8868.158999] ? apic_timer_interrupt+0xf/0x20 [ 8868.160992] [ 8868.162650] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8868.164817] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8868.167469] do_raw_spin_lock+0x177/0x190 [ 8868.169834] _raw_spin_lock+0x21/0x30 [ 8868.171626] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8868.174086] process_one_work+0x2c8/0x700 [ 8868.176436] worker_thread+0x296/0x6e0 [ 8868.177877] ? rescuer_thread+0x570/0x570 [ 8868.179398] kthread+0x1d1/0x200 [ 8868.181259] ? set_kthread_struct+0x70/0x70 [ 8868.183105] ret_from_fork+0x1f/0x30 [ 8872.056712] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 8872.062535] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8872.063550] Modules linked in: lustre(O) osp(O) ofd(O) [ 8872.077213] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) [ 8872.079628] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8872.079699] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8872.079707] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8872.079710] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8872.079803] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8872.079809] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8872.079814] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8872.079816] RDX: 00000000000071cd RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8872.079819] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8872.079821] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8872.079823] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8872.079826] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8872.079830] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8872.079832] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8872.079840] Call Trace: [ 8872.079855] [ 8872.079861] ? show_regs.cold.9+0x22/0x2f [ 8872.079872] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8872.079879] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8872.079884] ? watchdog+0x40/0x40 [ 8872.079888] ? hrtimer_interrupt+0x128/0x2c0 [ 8872.079902] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8872.079912] ? apic_timer_interrupt+0xf/0x20 [ 8872.079916] [ 8872.079920] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8872.079925] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8872.079930] do_raw_spin_lock+0x177/0x190 [ 8872.079935] _raw_spin_lock+0x21/0x30 [ 8872.079941] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8872.080278] obd_mod_free+0x46/0x140 [ptlrpc] [ 8872.080509] mdc_commit_open+0x91/0xb0 [mdc] [ 8872.080553] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8872.080940] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8872.081178] after_reply+0xe2c/0x1780 [ptlrpc] [ 8872.081374] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8872.081780] ? __next_timer_interrupt+0x160/0x160 [ 8872.081790] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8872.082032] ? do_wait_intr+0xf0/0xf0 [ 8872.082043] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8872.090270] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8872.090538] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8872.090546] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8872.090549] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8872.090565] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8872.090569] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8872.090581] RAX: 0000000000007efe RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8872.090583] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8872.090586] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8872.090588] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8872.090590] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8872.090594] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8872.090597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8872.090600] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8872.090608] Call Trace: [ 8872.090615] [ 8872.090622] ? show_regs.cold.9+0x22/0x2f [ 8872.090632] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8872.090640] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8872.090645] ? watchdog+0x40/0x40 [ 8872.090650] ? hrtimer_interrupt+0x128/0x2c0 [ 8872.090655] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8872.090667] ? apic_timer_interrupt+0xf/0x20 [ 8872.090672] [ 8872.090675] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8872.090681] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8872.090686] do_raw_spin_lock+0x177/0x190 [ 8872.090691] _raw_spin_lock+0x21/0x30 [ 8872.090698] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8872.124790] kthread+0x1d1/0x200 [ 8872.124812] ? set_kthread_struct+0x70/0x70 [ 8872.124817] ret_from_fork+0x1f/0x30 [ 8872.359727] ? do_raw_spin_unlock+0x75/0x190 [ 8872.362491] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8872.364747] ? do_wait_intr+0xf0/0xf0 [ 8872.365832] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8872.367787] mdc_reint+0x2d/0x110 [mdc] [ 8872.369630] mdc_unlink+0x658/0x870 [mdc] [ 8872.371545] lmv_unlink+0x34d/0xbd0 [lmv] [ 8872.373635] ll_unlink+0x24d/0x7f0 [lustre] [ 8872.375859] vfs_unlink+0x202/0x330 [ 8872.377652] do_unlinkat+0x2fa/0x460 [ 8872.379214] __x64_sys_unlink+0x37/0x50 [ 8872.381060] do_syscall_64+0xc1/0x440 [ 8872.382808] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8872.385403] RIP: 0033:0x7f83e5f5c10b [ 8872.387025] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8872.396414] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8872.399789] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8872.403075] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8872.406937] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8872.410031] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8872.413679] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8878.057149] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6364s! [ 8878.066947] Showing busy workqueues and worker pools: [ 8878.072811] workqueue events: flags=0x0 [ 8878.074492] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8878.081152] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8878.088918] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8878.095199] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8878.101854] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8878.108591] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8878.114371] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8878.116855] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8878.120409] workqueue events_long: flags=0x0 [ 8878.127538] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8878.135814] pending: lu_objects_flush [obdclass] [ 8878.138648] workqueue events_unbound: flags=0x2 [ 8878.144427] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8878.151055] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8878.157600] workqueue events_power_efficient: flags=0x80 [ 8878.160340] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8878.164720] pending: neigh_periodic_work, do_cache_clean [ 8878.170769] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8878.176901] pending: neigh_periodic_work, check_lifetime [ 8878.179693] workqueue mm_percpu_wq: flags=0x8 [ 8878.182175] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8878.188728] pending: vmstat_update, lru_add_drain_per_cpu [ 8878.191972] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8878.199649] pending: vmstat_update [ 8878.201711] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8878.205447] pending: lru_add_drain_per_cpu BAR(48) [ 8878.208016] workqueue ipv6_addrconf: flags=0x40008 [ 8878.209946] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8878.212723] pending: addrconf_verify_work [ 8878.214641] workqueue knbd0-recv: flags=0x1a [ 8878.216631] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8878.219716] in-flight: 650:recv_work [ 8878.221372] workqueue ptlrpc_pinger: flags=0x2 [ 8878.223221] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8878.225787] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8878.228850] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6365s workers=4 idle: 36 466 29757 [ 8878.232620] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8868s workers=2 idle: 652 [ 8881.128729] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 8881.131268] rcu: 0-....: (4371054 ticks this GP) idle=30a/1/0x4000000000000000 softirq=128519/128519 fqs=126395 [ 8881.137563] (detected by 1, t=6900169 jiffies, g=262441, q=59305) [ 8881.142014] Sending NMI from CPU 1 to CPUs 0: [ 8881.146906] NMI backtrace for cpu 0 [ 8881.146914] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8881.146916] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8881.146917] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8881.146919] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8881.146920] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 [ 8881.146923] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8881.146924] RDX: 0000000000005de9 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8881.146925] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8881.146927] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8881.146928] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8881.146930] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8881.146931] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8881.146932] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8881.146933] Call Trace: [ 8881.146934] [ 8881.146935] ? show_regs.cold.9+0x22/0x2f [ 8881.146937] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8881.146938] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8881.146939] ? nmi_handle+0xbd/0x210 [ 8881.146940] ? default_do_nmi+0x4b/0x1b0 [ 8881.146941] ? do_nmi+0x280/0x380 [ 8881.146942] ? end_repeat_nmi+0x16/0x1f [ 8881.146942] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8881.146944] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8881.146945] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8881.146947] [ 8881.146947] do_raw_spin_lock+0x177/0x190 [ 8881.146948] _raw_spin_lock+0x21/0x30 [ 8881.146949] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8881.146950] obd_mod_free+0x46/0x140 [ptlrpc] [ 8881.146952] mdc_commit_open+0x91/0xb0 [mdc] [ 8881.146953] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8881.146954] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8881.146956] after_reply+0xe2c/0x1780 [ptlrpc] [ 8881.146957] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8881.146958] ? __next_timer_interrupt+0x160/0x160 [ 8881.146959] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8881.146961] ? do_wait_intr+0xf0/0xf0 [ 8881.146962] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8881.146963] kthread+0x1d1/0x200 [ 8881.146964] ? set_kthread_struct+0x70/0x70 [ 8881.146965] ret_from_fork+0x1f/0x30 [ 8881.146996] rcu: rcu_sched kthread starved for 6369356 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 8881.343836] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8881.356262] rcu: RCU grace-period kthread stack dump: [ 8881.361869] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8881.368602] Call Trace: [ 8881.370304] __schedule+0x351/0xcb0 [ 8881.373588] schedule+0xc0/0x180 [ 8881.378880] schedule_timeout+0xb4/0x190 [ 8881.383562] ? do_raw_spin_unlock+0x75/0x190 [ 8881.387155] ? __next_timer_interrupt+0x160/0x160 [ 8881.392694] ? prepare_to_swait_event+0xf3/0x140 [ 8881.399430] rcu_gp_kthread+0x8dd/0x15b0 [ 8881.401789] ? force_qs_rnp+0x2c0/0x2c0 [ 8881.404268] kthread+0x1d1/0x200 [ 8881.405758] ? set_kthread_struct+0x70/0x70 [ 8881.409304] ret_from_fork+0x1f/0x30 [ 8881.411421] rcu: Stack dump where RCU GP kthread last ran: [ 8881.418455] Sending NMI from CPU 1 to CPUs 3: [ 8881.430582] NMI backtrace for cpu 3 [ 8881.431006] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8881.431008] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8881.431010] RIP: 0010:kvm_sched_clock_read+0x8/0x50 [ 8881.431011] Code: 10 48 85 c0 75 a2 48 83 05 a4 5f e6 02 01 b8 f4 ff ff ff c3 48 83 05 8e 5f e6 02 01 b8 f4 ff ff ff c3 48 83 05 b8 5a e6 02 01 <65> ff 05 d9 dc 38 56 48 83 05 b1 5a e6 02 01 65 48 8b 3d b1 e0 38 [ 8881.431014] RSP: 0018:ffffbc1d00d8cf88 EFLAGS: 00000002 [ 8881.431016] RAX: 0000000000000001 RBX: ffff9a61c219e3b0 RCX: 0000000000000000 [ 8881.431018] RDX: 0000000000000000 RSI: 0000000000010000 RDI: 0000000000000003 [ 8881.431019] RBP: ffff9a6190e827c0 R08: 0000000000000000 R09: 0000000000000000 [ 8881.431021] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000010000 [ 8881.431022] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8881.431024] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8881.431025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8881.431027] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8881.431028] Call Trace: [ 8881.431029] [ 8881.431030] ? show_regs.cold.9+0x22/0x2f [ 8881.431031] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8881.431032] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8881.431034] ? nmi_handle+0xbd/0x210 [ 8881.431035] ? default_do_nmi+0x4b/0x1b0 [ 8881.431036] ? do_nmi+0x280/0x380 [ 8881.431037] ? end_repeat_nmi+0x16/0x1f [ 8881.431039] ? kvm_sched_clock_read+0x8/0x50 [ 8881.431040] ? kvm_sched_clock_read+0x8/0x50 [ 8881.431041] ? kvm_sched_clock_read+0x8/0x50 [ 8881.431043] [ 8881.431044] [ 8881.431045] sched_clock+0xd/0x20 [ 8881.431046] sched_clock_cpu+0x24/0x150 [ 8881.431047] irqtime_account_irq+0x51/0x100 [ 8881.431047] irq_enter_rcu+0x81/0xf0 [ 8881.431048] irq_enter+0x1f/0x30 [ 8881.431049] smp_apic_timer_interrupt+0x24/0x270 [ 8881.431050] apic_timer_interrupt+0xf/0x20 [ 8881.431051] [ 8881.431052] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8881.431054] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8881.431056] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8881.431059] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8881.431060] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8881.431062] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8881.431063] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8881.431065] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8881.431066] kvm_wait+0xcd/0xe0 [ 8881.431067] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8881.431068] do_raw_spin_lock+0x177/0x190 [ 8881.431069] _raw_spin_lock+0x21/0x30 [ 8881.431070] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8881.431071] ? do_raw_spin_unlock+0x75/0x190 [ 8881.431072] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8881.431074] ? do_wait_intr+0xf0/0xf0 [ 8881.431075] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8881.431076] mdc_reint+0x2d/0x110 [mdc] [ 8881.431077] mdc_unlink+0x658/0x870 [mdc] [ 8881.431078] lmv_unlink+0x34d/0xbd0 [lmv] [ 8881.431079] ll_unlink+0x24d/0x7f0 [lustre] [ 8881.431081] vfs_unlink+0x202/0x330 [ 8881.431082] do_unlinkat+0x2fa/0x460 [ 8881.431083] __x64_sys_unlink+0x37/0x50 [ 8881.431084] do_syscall_64+0xc1/0x440 [ 8881.431085] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8881.431086] RIP: 0033:0x7f83e5f5c10b [ 8881.431088] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8881.431090] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8881.431093] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8881.431095] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8881.431096] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8881.431098] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8881.431099] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8896.060131] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 8896.064186] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8896.090822] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8896.096428] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8896.101617] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8896.106052] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8896.108824] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8896.117799] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8896.123096] RAX: 0000000000006c62 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8896.127026] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8896.130609] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8896.134944] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8896.137546] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8896.141199] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8896.146258] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8896.149736] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8896.154140] Call Trace: [ 8896.155678] [ 8896.156982] ? show_regs.cold.9+0x22/0x2f [ 8896.159551] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8896.162381] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8896.165531] ? watchdog+0x40/0x40 [ 8896.167175] ? hrtimer_interrupt+0x128/0x2c0 [ 8896.169621] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8896.172539] ? apic_timer_interrupt+0xf/0x20 [ 8896.175011] [ 8896.176167] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8896.179587] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8896.183124] do_raw_spin_lock+0x177/0x190 [ 8896.184934] _raw_spin_lock+0x21/0x30 [ 8896.187345] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8896.191634] process_one_work+0x2c8/0x700 [ 8896.193701] worker_thread+0x296/0x6e0 [ 8896.195798] ? rescuer_thread+0x570/0x570 [ 8896.198393] kthread+0x1d1/0x200 [ 8896.200154] ? set_kthread_struct+0x70/0x70 [ 8896.202325] ret_from_fork+0x1f/0x30 [ 8908.056057] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 8908.058347] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8908.059048] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 8908.062042] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 8908.062053] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8908.062102] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8908.062105] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8908.062107] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8908.062119] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8908.062122] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8908.062125] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8908.062126] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8908.062128] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8908.062130] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8908.062131] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8908.062134] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8908.062136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8908.062138] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8908.062144] Call Trace: [ 8908.062149] [ 8908.062153] ? show_regs.cold.9+0x22/0x2f [ 8908.062161] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8908.062166] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8908.062169] ? watchdog+0x40/0x40 [ 8908.062172] ? hrtimer_interrupt+0x128/0x2c0 [ 8908.062176] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8908.062183] ? apic_timer_interrupt+0xf/0x20 [ 8908.062188] [ 8908.062191] ? native_safe_halt+0x2e/0x50 [ 8908.062195] kvm_wait+0xcd/0xe0 [ 8908.062204] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8908.062210] do_raw_spin_lock+0x177/0x190 [ 8908.062215] _raw_spin_lock+0x21/0x30 [ 8908.062220] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8908.062515] ? do_raw_spin_unlock+0x75/0x190 [ 8908.062520] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8908.062725] ? do_wait_intr+0xf0/0xf0 [ 8908.062732] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8908.062883] mdc_reint+0x2d/0x110 [mdc] [ 8908.062915] mdc_unlink+0x658/0x870 [mdc] [ 8908.062940] lmv_unlink+0x34d/0xbd0 [lmv] [ 8908.062961] ll_unlink+0x24d/0x7f0 [lustre] [ 8908.063131] vfs_unlink+0x202/0x330 [ 8908.063137] do_unlinkat+0x2fa/0x460 [ 8908.063141] __x64_sys_unlink+0x37/0x50 [ 8908.063143] do_syscall_64+0xc1/0x440 [ 8908.063148] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8908.063152] RIP: 0033:0x7f83e5f5c10b [ 8908.063199] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8908.063204] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8908.063208] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8908.063211] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8908.063213] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8908.063215] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8908.063217] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8908.078464] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8908.078477] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8908.078480] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8908.078492] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8908.078496] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8908.078502] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8908.078504] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8908.078507] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8908.078509] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8908.078511] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8908.078514] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8908.078517] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8908.078519] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8908.078527] Call Trace: [ 8908.078531] [ 8908.078536] ? show_regs.cold.9+0x22/0x2f [ 8908.078554] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8908.078560] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8908.078564] ? watchdog+0x40/0x40 [ 8908.078568] ? hrtimer_interrupt+0x128/0x2c0 [ 8908.078572] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8908.078578] ? apic_timer_interrupt+0xf/0x20 [ 8908.078583] [ 8908.078585] ? native_safe_halt+0x2e/0x50 [ 8908.078589] kvm_wait+0xcd/0xe0 [ 8908.078596] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8908.078603] do_raw_spin_lock+0x177/0x190 [ 8908.078607] _raw_spin_lock+0x21/0x30 [ 8908.078612] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8908.083880] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8908.083987] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8908.083992] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8908.083995] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8908.084094] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8908.084097] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8908.084100] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8908.084102] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8908.084103] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8908.084105] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8908.084107] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8908.084109] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8908.084111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8908.084112] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8908.084119] Call Trace: [ 8908.084123] [ 8908.084127] ? show_regs.cold.9+0x22/0x2f [ 8908.084134] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8908.084139] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8908.084142] ? watchdog+0x40/0x40 [ 8908.084144] ? hrtimer_interrupt+0x128/0x2c0 [ 8908.084147] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8908.084152] ? apic_timer_interrupt+0xf/0x20 [ 8908.084155] [ 8908.084157] ? native_safe_halt+0x2e/0x50 [ 8908.084159] kvm_wait+0xcd/0xe0 [ 8908.084164] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8908.084170] do_raw_spin_lock+0x177/0x190 [ 8908.084173] _raw_spin_lock+0x21/0x30 [ 8908.084177] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8908.084437] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8908.084642] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8908.089271] obd_mod_free+0x46/0x140 [ptlrpc] [ 8908.089539] mdc_commit_open+0x91/0xb0 [mdc] [ 8908.110243] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8908.110534] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8908.110847] ? woken_wake_function+0x30/0x30 [ 8908.110856] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8908.118295] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8908.123318] kthread+0x1d1/0x200 [ 8908.123336] ? set_kthread_struct+0x70/0x70 [ 8908.123341] ret_from_fork+0x1f/0x30 [ 8908.389965] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8908.392604] after_reply+0xe2c/0x1780 [ptlrpc] [ 8908.394578] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8908.396904] ? __next_timer_interrupt+0x160/0x160 [ 8908.398875] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8908.400977] ? do_wait_intr+0xf0/0xf0 [ 8908.402432] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8908.404681] kthread+0x1d1/0x200 [ 8908.405841] ? set_kthread_struct+0x70/0x70 [ 8908.407518] ret_from_fork+0x1f/0x30 [ 8908.409571] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6395s! [ 8908.413230] Showing busy workqueues and worker pools: [ 8908.415481] workqueue events: flags=0x0 [ 8908.417359] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8908.421728] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8908.427361] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8908.430442] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8908.433233] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8908.436640] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8908.440446] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8908.442671] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8908.446822] workqueue events_long: flags=0x0 [ 8908.448464] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8908.451139] pending: lu_objects_flush [obdclass] [ 8908.453383] workqueue events_unbound: flags=0x2 [ 8908.455420] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8908.458552] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8908.461526] workqueue events_power_efficient: flags=0x80 [ 8908.464260] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8908.467547] pending: neigh_periodic_work, do_cache_clean [ 8908.470623] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8908.473842] pending: neigh_periodic_work, check_lifetime [ 8908.476689] workqueue mm_percpu_wq: flags=0x8 [ 8908.478813] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8908.481068] pending: vmstat_update, lru_add_drain_per_cpu [ 8908.482929] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8908.486025] pending: vmstat_update [ 8908.487422] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8908.489824] pending: lru_add_drain_per_cpu BAR(48) [ 8908.492333] workqueue ipv6_addrconf: flags=0x40008 [ 8908.494911] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8908.497598] pending: addrconf_verify_work [ 8908.499094] workqueue knbd0-recv: flags=0x1a [ 8908.500748] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8908.504324] in-flight: 650:recv_work [ 8908.506734] workqueue ptlrpc_pinger: flags=0x2 [ 8908.509149] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8908.512593] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8908.516468] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6395s workers=4 idle: 36 466 29757 [ 8908.519687] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8898s workers=2 idle: 652 [ 8924.060023] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 8924.063238] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8924.086923] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8924.093052] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8924.097145] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8924.100075] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8924.102195] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8924.110211] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8924.113261] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8924.116573] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8924.119896] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8924.123093] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8924.126384] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8924.129357] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8924.133287] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8924.135857] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8924.138579] Call Trace: [ 8924.139970] [ 8924.141194] ? show_regs.cold.9+0x22/0x2f [ 8924.143463] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8924.146178] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8924.148312] ? watchdog+0x40/0x40 [ 8924.149792] ? hrtimer_interrupt+0x128/0x2c0 [ 8924.151915] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8924.154149] ? apic_timer_interrupt+0xf/0x20 [ 8924.156559] [ 8924.157521] ? native_safe_halt+0x2e/0x50 [ 8924.159216] kvm_wait+0xcd/0xe0 [ 8924.160943] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8924.163071] do_raw_spin_lock+0x177/0x190 [ 8924.165213] _raw_spin_lock+0x21/0x30 [ 8924.170134] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8924.173039] process_one_work+0x2c8/0x700 [ 8924.174218] worker_thread+0x296/0x6e0 [ 8924.175335] ? rescuer_thread+0x570/0x570 [ 8924.177082] kthread+0x1d1/0x200 [ 8924.178291] ? set_kthread_struct+0x70/0x70 [ 8924.179708] ret_from_fork+0x1f/0x30 [ 8936.056053] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 8936.059021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 8936.059302] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8936.059372] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8936.059377] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8936.059380] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8936.059394] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8936.059398] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8936.059404] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8936.062025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8936.062038] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8936.062106] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8936.062111] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8936.062114] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8936.062128] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8936.062132] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8936.062137] RAX: 000000000000758a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8936.062140] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8936.062142] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8936.062144] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8936.062147] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8936.062150] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8936.062152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8936.062155] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8936.062163] Call Trace: [ 8936.062167] [ 8936.062172] ? show_regs.cold.9+0x22/0x2f [ 8936.062182] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8936.062189] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8936.062192] ? watchdog+0x40/0x40 [ 8936.062197] ? hrtimer_interrupt+0x128/0x2c0 [ 8936.062201] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8936.062211] ? apic_timer_interrupt+0xf/0x20 [ 8936.062216] [ 8936.062219] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8936.062224] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8936.062229] do_raw_spin_lock+0x177/0x190 [ 8936.062234] _raw_spin_lock+0x21/0x30 [ 8936.062240] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8936.063188] ? do_raw_spin_unlock+0x75/0x190 [ 8936.063195] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8936.064089] ? do_wait_intr+0xf0/0xf0 [ 8936.064097] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8936.064364] mdc_reint+0x2d/0x110 [mdc] [ 8936.064403] mdc_unlink+0x658/0x870 [mdc] [ 8936.064433] lmv_unlink+0x34d/0xbd0 [lmv] [ 8936.064462] ll_unlink+0x24d/0x7f0 [lustre] [ 8936.064578] vfs_unlink+0x202/0x330 [ 8936.064586] do_unlinkat+0x2fa/0x460 [ 8936.064590] __x64_sys_unlink+0x37/0x50 [ 8936.064593] do_syscall_64+0xc1/0x440 [ 8936.064600] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8936.064607] RIP: 0033:0x7f83e5f5c10b [ 8936.064614] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8936.064618] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8936.064623] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8936.064626] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8936.064628] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8936.064630] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8936.064632] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8936.069097] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8936.069248] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8936.069255] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8936.069266] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 8936.069288] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 8936.069292] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8936.069297] RAX: 00000000000056f6 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8936.069299] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8936.069301] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8936.069304] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8936.069306] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8936.069309] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8936.069311] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8936.069313] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8936.093886] RDX: 00000000000070ea RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8936.093896] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8936.093898] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8936.093901] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8936.093903] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8936.093906] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8936.093909] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8936.093917] Call Trace: [ 8936.093923] [ 8936.093928] ? show_regs.cold.9+0x22/0x2f [ 8936.093942] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8936.093948] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8936.093953] ? watchdog+0x40/0x40 [ 8936.093958] ? hrtimer_interrupt+0x128/0x2c0 [ 8936.093962] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8936.093969] ? apic_timer_interrupt+0xf/0x20 [ 8936.093975] [ 8936.093977] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8936.093984] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8936.093989] do_raw_spin_lock+0x177/0x190 [ 8936.093994] _raw_spin_lock+0x21/0x30 [ 8936.094000] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8936.108627] Call Trace: [ 8936.115904] obd_mod_free+0x46/0x140 [ptlrpc] [ 8936.116769] mdc_commit_open+0x91/0xb0 [mdc] [ 8936.116805] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8936.117004] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8936.117226] after_reply+0xe2c/0x1780 [ptlrpc] [ 8936.117440] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8936.125169] [ 8936.125183] ? show_regs.cold.9+0x22/0x2f [ 8936.125203] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8936.125209] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8936.125213] ? watchdog+0x40/0x40 [ 8936.125217] ? hrtimer_interrupt+0x128/0x2c0 [ 8936.125221] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8936.125230] ? apic_timer_interrupt+0xf/0x20 [ 8936.125235] [ 8936.125238] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 8936.134302] ? __next_timer_interrupt+0x160/0x160 [ 8936.134324] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8936.134632] ? do_wait_intr+0xf0/0xf0 [ 8936.134642] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8936.134847] kthread+0x1d1/0x200 [ 8936.134852] ? set_kthread_struct+0x70/0x70 [ 8936.134856] ret_from_fork+0x1f/0x30 [ 8936.714538] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8936.717880] do_raw_spin_lock+0x177/0x190 [ 8936.720539] _raw_spin_lock+0x21/0x30 [ 8936.722141] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8936.729195] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8936.731229] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8936.735920] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8936.740206] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8936.745751] ? woken_wake_function+0x30/0x30 [ 8936.750364] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8936.753765] kthread+0x1d1/0x200 [ 8936.755398] ? set_kthread_struct+0x70/0x70 [ 8936.757288] ret_from_fork+0x1f/0x30 [ 8938.982009] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6425s! [ 8938.987571] Showing busy workqueues and worker pools: [ 8938.991888] workqueue events: flags=0x0 [ 8938.994312] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8938.999995] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8939.007329] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8939.011926] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8939.017519] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8939.021922] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8939.027270] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8939.032698] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8939.041011] workqueue events_long: flags=0x0 [ 8939.043976] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8939.048672] pending: lu_objects_flush [obdclass] [ 8939.051569] workqueue events_unbound: flags=0x2 [ 8939.055600] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8939.059565] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8939.062751] workqueue events_power_efficient: flags=0x80 [ 8939.066941] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8939.071577] pending: neigh_periodic_work, do_cache_clean [ 8939.075581] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8939.079720] pending: neigh_periodic_work, check_lifetime [ 8939.081763] workqueue mm_percpu_wq: flags=0x8 [ 8939.085016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8939.087176] pending: vmstat_update, lru_add_drain_per_cpu [ 8939.090919] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8939.095199] pending: vmstat_update [ 8939.098038] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8939.102258] pending: lru_add_drain_per_cpu BAR(48) [ 8939.105995] workqueue ipv6_addrconf: flags=0x40008 [ 8939.108717] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8939.112346] pending: addrconf_verify_work [ 8939.115679] workqueue knbd0-recv: flags=0x1a [ 8939.118745] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8939.123005] in-flight: 650:recv_work [ 8939.125880] workqueue ptlrpc_pinger: flags=0x2 [ 8939.129036] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8939.133938] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8939.137752] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6425s workers=4 idle: 36 466 29757 [ 8939.142350] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8929s workers=2 idle: 652 [ 8952.060026] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 8952.065510] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8952.095390] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8952.100967] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8952.105629] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8952.109283] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8952.111537] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8952.121603] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8952.126503] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8952.131181] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8952.134150] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8952.137845] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8952.141635] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8952.146034] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8952.154936] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8952.159487] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8952.164760] Call Trace: [ 8952.166925] [ 8952.168760] ? show_regs.cold.9+0x22/0x2f [ 8952.171251] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8952.175021] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8952.179553] ? watchdog+0x40/0x40 [ 8952.183188] ? hrtimer_interrupt+0x128/0x2c0 [ 8952.185755] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8952.188564] ? apic_timer_interrupt+0xf/0x20 [ 8952.191508] [ 8952.192579] ? native_safe_halt+0x2e/0x50 [ 8952.195071] kvm_wait+0xcd/0xe0 [ 8952.196839] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8952.199670] do_raw_spin_lock+0x177/0x190 [ 8952.201739] _raw_spin_lock+0x21/0x30 [ 8952.203502] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8952.206413] process_one_work+0x2c8/0x700 [ 8952.208562] worker_thread+0x296/0x6e0 [ 8952.210246] ? rescuer_thread+0x570/0x570 [ 8952.212585] kthread+0x1d1/0x200 [ 8952.214052] ? set_kthread_struct+0x70/0x70 [ 8952.216014] ret_from_fork+0x1f/0x30 [ 8964.056089] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 8964.059030] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 8964.060583] Modules linked in: [ 8964.062030] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8964.062038] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8964.062103] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8964.062109] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8964.062112] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8964.062123] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8964.062126] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8964.062131] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8964.062133] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8964.062135] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8964.062137] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8964.062139] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8964.062142] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8964.062339] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8964.062342] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8964.062349] Call Trace: [ 8964.062354] [ 8964.062359] ? show_regs.cold.9+0x22/0x2f [ 8964.062368] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8964.062375] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8964.062379] ? watchdog+0x40/0x40 [ 8964.062383] ? hrtimer_interrupt+0x128/0x2c0 [ 8964.062387] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8964.062393] ? apic_timer_interrupt+0xf/0x20 [ 8964.062398] [ 8964.062400] ? native_safe_halt+0x2e/0x50 [ 8964.062404] kvm_wait+0xcd/0xe0 [ 8964.062410] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8964.062416] do_raw_spin_lock+0x177/0x190 [ 8964.062420] _raw_spin_lock+0x21/0x30 [ 8964.062425] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8964.062678] ? do_raw_spin_unlock+0x75/0x190 [ 8964.062684] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8964.062875] ? do_wait_intr+0xf0/0xf0 [ 8964.062884] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8964.063091] mdc_reint+0x2d/0x110 [mdc] [ 8964.063128] mdc_unlink+0x658/0x870 [mdc] [ 8964.063165] lmv_unlink+0x34d/0xbd0 [lmv] [ 8964.063409] ll_unlink+0x24d/0x7f0 [lustre] [ 8964.063541] vfs_unlink+0x202/0x330 [ 8964.063550] do_unlinkat+0x2fa/0x460 [ 8964.063592] __x64_sys_unlink+0x37/0x50 [ 8964.063598] do_syscall_64+0xc1/0x440 [ 8964.063605] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8964.063611] RIP: 0033:0x7f83e5f5c10b [ 8964.063617] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8964.063621] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8964.063625] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8964.063627] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8964.063629] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8964.063631] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8964.063633] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8964.072623] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8964.072709] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8964.072717] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8964.072720] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8964.072733] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8964.072737] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8964.072741] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8964.072744] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8964.072746] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8964.072748] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8964.072750] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8964.072754] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8964.072757] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8964.072759] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8964.072767] Call Trace: [ 8964.072773] [ 8964.072779] ? show_regs.cold.9+0x22/0x2f [ 8964.072791] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8964.072798] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8964.072809] ? watchdog+0x40/0x40 [ 8964.072814] ? hrtimer_interrupt+0x128/0x2c0 [ 8964.072818] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8964.072825] ? apic_timer_interrupt+0xf/0x20 [ 8964.072830] [ 8964.072833] ? native_safe_halt+0x2e/0x50 [ 8964.074717] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8964.074894] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8964.074901] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8964.074903] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8964.074912] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8964.074915] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8964.074919] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8964.074921] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8964.074923] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8964.074925] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8964.074987] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8964.074992] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8964.074995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8964.074997] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8964.075012] Call Trace: [ 8964.075016] [ 8964.075020] ? show_regs.cold.9+0x22/0x2f [ 8964.075031] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8964.075037] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8964.075041] ? watchdog+0x40/0x40 [ 8964.075046] ? hrtimer_interrupt+0x128/0x2c0 [ 8964.075050] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8964.075056] ? apic_timer_interrupt+0xf/0x20 [ 8964.075061] [ 8964.075064] ? native_safe_halt+0x2e/0x50 [ 8964.075068] kvm_wait+0xcd/0xe0 [ 8964.075075] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8964.075082] do_raw_spin_lock+0x177/0x190 [ 8964.075086] _raw_spin_lock+0x21/0x30 [ 8964.075091] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8964.075476] obd_mod_free+0x46/0x140 [ptlrpc] [ 8964.075674] mdc_commit_open+0x91/0xb0 [mdc] [ 8964.075712] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8964.076432] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8964.076650] after_reply+0xe2c/0x1780 [ptlrpc] [ 8964.077150] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8964.077570] ? __next_timer_interrupt+0x160/0x160 [ 8964.077581] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8964.086487] kvm_wait+0xcd/0xe0 [ 8964.086510] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8964.086522] do_raw_spin_lock+0x177/0x190 [ 8964.086527] _raw_spin_lock+0x21/0x30 [ 8964.086538] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8964.086864] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8964.087504] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8964.087723] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8964.161748] ? do_wait_intr+0xf0/0xf0 [ 8964.161772] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8964.162232] kthread+0x1d1/0x200 [ 8964.162239] ? set_kthread_struct+0x70/0x70 [ 8964.162349] ret_from_fork+0x1f/0x30 [ 8964.643448] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8964.646595] ? woken_wake_function+0x30/0x30 [ 8964.649219] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8964.652680] kthread+0x1d1/0x200 [ 8964.653910] ? set_kthread_struct+0x70/0x70 [ 8964.655643] ret_from_fork+0x1f/0x30 [ 8969.184164] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6455s! [ 8969.188656] Showing busy workqueues and worker pools: [ 8969.191024] workqueue events: flags=0x0 [ 8969.192725] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8969.195318] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8969.199470] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8969.203122] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8969.205823] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8969.209016] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8969.212024] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8969.218448] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8969.228231] workqueue events_long: flags=0x0 [ 8969.231271] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8969.237087] pending: lu_objects_flush [obdclass] [ 8969.246518] workqueue events_unbound: flags=0x2 [ 8969.247969] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8969.257415] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8969.262542] workqueue events_power_efficient: flags=0x80 [ 8969.268113] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8969.271788] pending: neigh_periodic_work, do_cache_clean [ 8969.273914] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8969.276746] pending: neigh_periodic_work, check_lifetime [ 8969.279898] workqueue mm_percpu_wq: flags=0x8 [ 8969.281543] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8969.283777] pending: vmstat_update, lru_add_drain_per_cpu [ 8969.286574] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8969.291947] pending: vmstat_update [ 8969.294744] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8969.301046] pending: lru_add_drain_per_cpu BAR(48) [ 8969.307517] workqueue ipv6_addrconf: flags=0x40008 [ 8969.310552] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8969.318282] pending: addrconf_verify_work [ 8969.322585] workqueue knbd0-recv: flags=0x1a [ 8969.325783] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8969.331279] in-flight: 650:recv_work [ 8969.336476] workqueue ptlrpc_pinger: flags=0x2 [ 8969.339706] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8969.345394] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8969.350371] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6456s workers=4 idle: 36 466 29757 [ 8969.354717] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8959s workers=2 idle: 652 [ 8980.060023] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 8980.062514] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8980.087558] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8980.092167] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8980.096439] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8980.100105] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8980.101934] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8980.110301] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8980.113626] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8980.117788] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 8980.121698] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 8980.124711] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 8980.127525] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 8980.131027] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 8980.136032] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8980.140420] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 8980.142602] Call Trace: [ 8980.143639] [ 8980.144229] ? show_regs.cold.9+0x22/0x2f [ 8980.145309] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8980.147123] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8980.148772] ? watchdog+0x40/0x40 [ 8980.150011] ? hrtimer_interrupt+0x128/0x2c0 [ 8980.151500] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8980.153772] ? apic_timer_interrupt+0xf/0x20 [ 8980.155249] [ 8980.155875] ? native_safe_halt+0x2e/0x50 [ 8980.157193] kvm_wait+0xcd/0xe0 [ 8980.158300] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8980.159727] do_raw_spin_lock+0x177/0x190 [ 8980.161299] _raw_spin_lock+0x21/0x30 [ 8980.162444] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8980.164392] process_one_work+0x2c8/0x700 [ 8980.165618] worker_thread+0x296/0x6e0 [ 8980.166673] ? rescuer_thread+0x570/0x570 [ 8980.168179] kthread+0x1d1/0x200 [ 8980.169245] ? set_kthread_struct+0x70/0x70 [ 8980.170600] ret_from_fork+0x1f/0x30 [ 8992.059050] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 8992.061180] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8992.062061] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 8992.083435] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8992.083451] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8992.083455] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8992.083469] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8992.083473] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8992.083478] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8992.083481] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 8992.083483] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8992.083485] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 8992.083487] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 8992.083491] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 8992.083494] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8992.083497] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 8992.083508] Call Trace: [ 8992.083514] [ 8992.083520] ? show_regs.cold.9+0x22/0x2f [ 8992.083530] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8992.083538] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8992.083543] ? watchdog+0x40/0x40 [ 8992.083548] ? hrtimer_interrupt+0x128/0x2c0 [ 8992.083552] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8992.083565] ? apic_timer_interrupt+0xf/0x20 [ 8992.083570] [ 8992.083573] ? native_safe_halt+0x2e/0x50 [ 8992.083577] kvm_wait+0xcd/0xe0 [ 8992.083585] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8992.083595] do_raw_spin_lock+0x177/0x190 [ 8992.083600] _raw_spin_lock+0x21/0x30 [ 8992.083605] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8992.083908] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8992.084305] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8992.084603] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8992.084845] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8992.085135] ? woken_wake_function+0x30/0x30 [ 8992.085146] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8992.085487] kthread+0x1d1/0x200 [ 8992.085493] ? set_kthread_struct+0x70/0x70 [ 8992.085497] ret_from_fork+0x1f/0x30 [ 8992.236402] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8992.267580] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8992.272411] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8992.276182] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8992.278170] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8992.285348] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8992.288266] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 8992.291309] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 8992.294440] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8992.297364] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 8992.302531] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 8992.306639] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 8992.314411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8992.319426] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 8992.322477] Call Trace: [ 8992.323547] [ 8992.324530] ? show_regs.cold.9+0x22/0x2f [ 8992.326647] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8992.329562] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8992.332649] ? watchdog+0x40/0x40 [ 8992.334670] ? hrtimer_interrupt+0x128/0x2c0 [ 8992.337481] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8992.341671] ? apic_timer_interrupt+0xf/0x20 [ 8992.344183] [ 8992.345667] ? native_safe_halt+0x2e/0x50 [ 8992.347737] kvm_wait+0xcd/0xe0 [ 8992.352401] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8992.355160] do_raw_spin_lock+0x177/0x190 [ 8992.358067] _raw_spin_lock+0x21/0x30 [ 8992.361714] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8992.370196] ? do_raw_spin_unlock+0x75/0x190 [ 8992.376744] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8992.383954] ? do_wait_intr+0xf0/0xf0 [ 8992.386412] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8992.393907] mdc_reint+0x2d/0x110 [mdc] [ 8992.399870] mdc_unlink+0x658/0x870 [mdc] [ 8992.403503] lmv_unlink+0x34d/0xbd0 [lmv] [ 8992.406562] ll_unlink+0x24d/0x7f0 [lustre] [ 8992.413910] vfs_unlink+0x202/0x330 [ 8992.420200] do_unlinkat+0x2fa/0x460 [ 8992.422656] __x64_sys_unlink+0x37/0x50 [ 8992.426166] do_syscall_64+0xc1/0x440 [ 8992.431208] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8992.439586] RIP: 0033:0x7f83e5f5c10b [ 8992.445380] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 8992.471721] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 8992.485173] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 8992.489519] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 8992.493966] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 8992.497233] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 8992.501247] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 8996.056427] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 8996.059128] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8996.110494] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8996.125301] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8996.130664] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8996.134409] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8996.146538] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8996.157639] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 8996.159584] RDX: 00000000000001e4 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 8996.166364] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 8996.171328] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 8996.176543] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8996.181363] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 8996.187113] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8996.192069] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 8996.199762] Call Trace: [ 8996.202222] [ 8996.203253] ? show_regs.cold.9+0x22/0x2f [ 8996.208388] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8996.212200] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8996.215428] ? watchdog+0x40/0x40 [ 8996.218428] ? hrtimer_interrupt+0x128/0x2c0 [ 8996.224875] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8996.233952] ? apic_timer_interrupt+0xf/0x20 [ 8996.242112] [ 8996.246549] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8996.257034] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8996.260337] do_raw_spin_lock+0x177/0x190 [ 8996.263502] _raw_spin_lock+0x21/0x30 [ 8996.268186] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8996.273413] obd_mod_free+0x46/0x140 [ptlrpc] [ 8996.276179] mdc_commit_open+0x91/0xb0 [mdc] [ 8996.283128] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8996.286290] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8996.294321] after_reply+0xe2c/0x1780 [ptlrpc] [ 8996.302103] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8996.306581] ? __next_timer_interrupt+0x160/0x160 [ 8996.311964] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8996.314453] ? do_wait_intr+0xf0/0xf0 [ 8996.317199] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8996.324182] kthread+0x1d1/0x200 [ 8996.328067] ? set_kthread_struct+0x70/0x70 [ 8996.330864] ret_from_fork+0x1f/0x30 [ 8999.395206] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6486s! [ 8999.403726] Showing busy workqueues and worker pools: [ 8999.409176] workqueue events: flags=0x0 [ 8999.411194] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8999.415631] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 8999.424279] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8999.428398] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 8999.435296] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8999.439786] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 8999.443947] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8999.447363] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 8999.454995] workqueue events_long: flags=0x0 [ 8999.457015] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8999.461844] pending: lu_objects_flush [obdclass] [ 8999.466146] workqueue events_unbound: flags=0x2 [ 8999.468953] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8999.475753] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8999.481024] workqueue events_power_efficient: flags=0x80 [ 8999.483359] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8999.490816] pending: neigh_periodic_work, do_cache_clean [ 8999.493847] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8999.501224] pending: neigh_periodic_work, check_lifetime [ 8999.506342] workqueue mm_percpu_wq: flags=0x8 [ 8999.515685] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8999.523453] pending: vmstat_update, lru_add_drain_per_cpu [ 8999.528541] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8999.533548] pending: vmstat_update [ 8999.536123] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 8999.541751] pending: lru_add_drain_per_cpu BAR(48) [ 8999.543627] workqueue ipv6_addrconf: flags=0x40008 [ 8999.553357] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8999.562825] pending: addrconf_verify_work [ 8999.565035] workqueue knbd0-recv: flags=0x1a [ 8999.568977] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8999.572965] in-flight: 650:recv_work [ 8999.575395] workqueue ptlrpc_pinger: flags=0x2 [ 8999.578412] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8999.582083] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 8999.587424] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6486s workers=4 idle: 36 466 29757 [ 8999.595062] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8989s workers=2 idle: 652 [ 9008.060029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9008.063117] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9008.089516] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9008.093662] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9008.107193] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9008.117031] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9008.122136] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9008.131124] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9008.134475] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9008.137332] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9008.141594] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9008.143791] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9008.150181] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9008.158298] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9008.170176] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9008.177652] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9008.182580] Call Trace: [ 9008.183476] [ 9008.186109] ? show_regs.cold.9+0x22/0x2f [ 9008.191253] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9008.195205] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9008.202950] ? watchdog+0x40/0x40 [ 9008.204187] ? hrtimer_interrupt+0x128/0x2c0 [ 9008.207867] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9008.212163] ? apic_timer_interrupt+0xf/0x20 [ 9008.215086] [ 9008.218473] ? native_safe_halt+0x2e/0x50 [ 9008.223251] kvm_wait+0xcd/0xe0 [ 9008.227093] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9008.231319] do_raw_spin_lock+0x177/0x190 [ 9008.238009] _raw_spin_lock+0x21/0x30 [ 9008.241994] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9008.248559] process_one_work+0x2c8/0x700 [ 9008.250865] worker_thread+0x296/0x6e0 [ 9008.253972] ? rescuer_thread+0x570/0x570 [ 9008.258702] kthread+0x1d1/0x200 [ 9008.260344] ? set_kthread_struct+0x70/0x70 [ 9008.263102] ret_from_fork+0x1f/0x30 [ 9020.059021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9020.061240] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9020.067439] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 9020.079550] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9020.079565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9020.079568] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9020.079583] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9020.079587] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9020.079593] RAX: 0000000000003f2a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9020.079596] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9020.079598] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9020.079601] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9020.079604] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9020.079607] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9020.079610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9020.079612] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9020.079622] Call Trace: [ 9020.079627] [ 9020.079634] ? show_regs.cold.9+0x22/0x2f [ 9020.079727] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9020.079735] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9020.079739] ? watchdog+0x40/0x40 [ 9020.079744] ? hrtimer_interrupt+0x128/0x2c0 [ 9020.079748] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9020.079756] ? apic_timer_interrupt+0xf/0x20 [ 9020.079760] [ 9020.079763] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9020.079768] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9020.079774] do_raw_spin_lock+0x177/0x190 [ 9020.079779] _raw_spin_lock+0x21/0x30 [ 9020.079783] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9020.080055] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9020.080251] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9020.080576] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9020.080804] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9020.081022] ? woken_wake_function+0x30/0x30 [ 9020.081031] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9020.081249] kthread+0x1d1/0x200 [ 9020.081256] ? set_kthread_struct+0x70/0x70 [ 9020.081259] ret_from_fork+0x1f/0x30 [ 9020.203976] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9020.223609] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9020.227830] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9020.231139] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9020.233530] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9020.240167] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9020.242807] RAX: 000000000000780e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9020.245298] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9020.248057] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9020.250561] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9020.254020] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9020.258891] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9020.263588] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9020.266820] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9020.271959] Call Trace: [ 9020.273587] [ 9020.274549] ? show_regs.cold.9+0x22/0x2f [ 9020.276237] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9020.279159] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9020.282579] ? watchdog+0x40/0x40 [ 9020.284040] ? hrtimer_interrupt+0x128/0x2c0 [ 9020.286539] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9020.289391] ? apic_timer_interrupt+0xf/0x20 [ 9020.291778] [ 9020.292815] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9020.295784] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9020.299357] do_raw_spin_lock+0x177/0x190 [ 9020.301865] _raw_spin_lock+0x21/0x30 [ 9020.304592] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9020.307944] ? do_raw_spin_unlock+0x75/0x190 [ 9020.310812] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9020.314225] ? do_wait_intr+0xf0/0xf0 [ 9020.316553] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9020.319340] mdc_reint+0x2d/0x110 [mdc] [ 9020.322508] mdc_unlink+0x658/0x870 [mdc] [ 9020.324593] lmv_unlink+0x34d/0xbd0 [lmv] [ 9020.326820] ll_unlink+0x24d/0x7f0 [lustre] [ 9020.329162] vfs_unlink+0x202/0x330 [ 9020.330649] do_unlinkat+0x2fa/0x460 [ 9020.332195] __x64_sys_unlink+0x37/0x50 [ 9020.333803] do_syscall_64+0xc1/0x440 [ 9020.335310] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9020.337433] RIP: 0033:0x7f83e5f5c10b [ 9020.339032] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9020.346601] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9020.349838] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9020.352782] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9020.355638] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9020.358748] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9020.362253] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9024.059614] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 9024.066947] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9024.106023] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9024.112017] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9024.115885] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9024.118825] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9024.141819] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9024.150321] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9024.155197] RDX: 00000000000028d5 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9024.159216] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9024.164988] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9024.168928] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9024.172458] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9024.176208] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9024.179443] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9024.183108] Call Trace: [ 9024.184280] [ 9024.185398] ? show_regs.cold.9+0x22/0x2f [ 9024.190033] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9024.193770] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9024.195965] ? watchdog+0x40/0x40 [ 9024.198938] ? hrtimer_interrupt+0x128/0x2c0 [ 9024.200900] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9024.204850] ? apic_timer_interrupt+0xf/0x20 [ 9024.206762] [ 9024.207744] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9024.210526] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9024.213157] do_raw_spin_lock+0x177/0x190 [ 9024.215502] _raw_spin_lock+0x21/0x30 [ 9024.217083] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9024.222118] obd_mod_free+0x46/0x140 [ptlrpc] [ 9024.225470] mdc_commit_open+0x91/0xb0 [mdc] [ 9024.227944] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9024.230292] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9024.233856] after_reply+0xe2c/0x1780 [ptlrpc] [ 9024.236085] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9024.238869] ? __next_timer_interrupt+0x160/0x160 [ 9024.240551] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9024.242943] ? do_wait_intr+0xf0/0xf0 [ 9024.244934] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9024.247695] kthread+0x1d1/0x200 [ 9024.249178] ? set_kthread_struct+0x70/0x70 [ 9024.251543] ret_from_fork+0x1f/0x30 [ 9029.602053] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6516s! [ 9029.606159] Showing busy workqueues and worker pools: [ 9029.608922] workqueue events: flags=0x0 [ 9029.610973] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9029.614065] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9029.617649] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9029.620736] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9029.623816] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9029.627132] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9029.630935] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9029.634315] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9029.637553] workqueue events_long: flags=0x0 [ 9029.638627] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9029.650862] pending: lu_objects_flush [obdclass] [ 9029.653429] workqueue events_unbound: flags=0x2 [ 9029.657078] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9029.659283] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9029.663673] workqueue events_power_efficient: flags=0x80 [ 9029.666557] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9029.668810] pending: neigh_periodic_work, do_cache_clean [ 9029.670492] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9029.672226] pending: neigh_periodic_work, check_lifetime [ 9029.676024] workqueue mm_percpu_wq: flags=0x8 [ 9029.677879] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9029.680258] pending: vmstat_update, lru_add_drain_per_cpu [ 9029.683332] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9029.685156] pending: vmstat_update [ 9029.687066] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9029.689194] pending: lru_add_drain_per_cpu BAR(48) [ 9029.693087] workqueue ipv6_addrconf: flags=0x40008 [ 9029.699241] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9029.706287] pending: addrconf_verify_work [ 9029.709348] workqueue knbd0-recv: flags=0x1a [ 9029.713173] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9029.716095] in-flight: 650:recv_work [ 9029.720550] workqueue ptlrpc_pinger: flags=0x2 [ 9029.725567] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9029.727198] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9029.732208] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6516s workers=4 idle: 36 466 29757 [ 9029.735765] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9019s workers=2 idle: 652 [ 9036.060139] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9036.072567] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9036.127474] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9036.135073] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9036.146824] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9036.154997] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9036.159868] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9036.179439] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9036.188888] RAX: 0000000000001d25 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9036.195049] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9036.198117] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9036.212246] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9036.229394] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9036.246781] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9036.255939] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9036.262882] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9036.267060] Call Trace: [ 9036.273058] [ 9036.274109] ? show_regs.cold.9+0x22/0x2f [ 9036.287627] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9036.297695] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9036.306058] ? watchdog+0x40/0x40 [ 9036.310565] ? hrtimer_interrupt+0x128/0x2c0 [ 9036.314341] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9036.322531] ? apic_timer_interrupt+0xf/0x20 [ 9036.326267] [ 9036.328065] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9036.337030] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9036.338888] do_raw_spin_lock+0x177/0x190 [ 9036.348958] _raw_spin_lock+0x21/0x30 [ 9036.354396] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9036.360424] process_one_work+0x2c8/0x700 [ 9036.366000] worker_thread+0x296/0x6e0 [ 9036.374028] ? rescuer_thread+0x570/0x570 [ 9036.376420] kthread+0x1d1/0x200 [ 9036.388875] ? set_kthread_struct+0x70/0x70 [ 9036.393381] ret_from_fork+0x1f/0x30 [ 9048.059445] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9048.063235] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 9048.063534] Modules linked in: [ 9048.065663] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9048.065735] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9048.065743] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9048.065746] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9048.065760] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9048.065764] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9048.065768] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9048.065771] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9048.065773] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9048.065775] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9048.065778] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9048.065781] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9048.065783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9048.065785] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9048.065794] Call Trace: [ 9048.065800] [ 9048.065805] ? show_regs.cold.9+0x22/0x2f [ 9048.065815] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9048.065821] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9048.065826] ? watchdog+0x40/0x40 [ 9048.065831] ? hrtimer_interrupt+0x128/0x2c0 [ 9048.065835] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9048.065842] ? apic_timer_interrupt+0xf/0x20 [ 9048.065847] [ 9048.065849] ? native_safe_halt+0x2e/0x50 [ 9048.065853] kvm_wait+0xcd/0xe0 [ 9048.065860] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9048.065867] do_raw_spin_lock+0x177/0x190 [ 9048.065872] _raw_spin_lock+0x21/0x30 [ 9048.065877] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9048.066141] ? do_raw_spin_unlock+0x75/0x190 [ 9048.066147] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9048.066352] ? do_wait_intr+0xf0/0xf0 [ 9048.066360] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9048.066554] mdc_reint+0x2d/0x110 [mdc] [ 9048.066591] mdc_unlink+0x658/0x870 [mdc] [ 9048.066619] lmv_unlink+0x34d/0xbd0 [lmv] [ 9048.066650] ll_unlink+0x24d/0x7f0 [lustre] [ 9048.066766] vfs_unlink+0x202/0x330 [ 9048.066773] do_unlinkat+0x2fa/0x460 [ 9048.066777] __x64_sys_unlink+0x37/0x50 [ 9048.066781] do_syscall_64+0xc1/0x440 [ 9048.066787] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9048.066792] RIP: 0033:0x7f83e5f5c10b [ 9048.066799] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9048.066802] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9048.066807] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9048.066810] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9048.066812] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9048.066814] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9048.066816] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9048.521706] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9048.587716] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9048.612464] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9048.627567] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9048.635306] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9048.658635] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9048.671288] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9048.682707] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9048.690129] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9048.699861] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9048.710964] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9048.730673] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9048.733668] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9048.755936] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9048.778077] Call Trace: [ 9048.778992] [ 9048.779729] ? show_regs.cold.9+0x22/0x2f [ 9048.792663] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9048.796328] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9048.801263] ? watchdog+0x40/0x40 [ 9048.802616] ? hrtimer_interrupt+0x128/0x2c0 [ 9048.810033] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9048.821992] ? apic_timer_interrupt+0xf/0x20 [ 9048.828301] [ 9048.832887] ? native_safe_halt+0x2e/0x50 [ 9048.834294] kvm_wait+0xcd/0xe0 [ 9048.835462] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9048.850674] do_raw_spin_lock+0x177/0x190 [ 9048.854735] _raw_spin_lock+0x21/0x30 [ 9048.856484] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9048.862349] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9048.867274] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9048.869869] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9048.872885] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9048.875785] ? woken_wake_function+0x30/0x30 [ 9048.877938] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9048.880966] kthread+0x1d1/0x200 [ 9048.882500] ? set_kthread_struct+0x70/0x70 [ 9048.884444] ret_from_fork+0x1f/0x30 [ 9056.062094] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 9056.065364] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9056.105993] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9056.121002] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9056.124698] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9056.131875] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9056.154726] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9056.158610] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9056.163322] RDX: 0000000000007d6e RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9056.172464] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9056.179071] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9056.181103] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9056.191455] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9056.198119] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9056.200540] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9056.203998] Call Trace: [ 9056.205201] [ 9056.206388] ? show_regs.cold.9+0x22/0x2f [ 9056.208050] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9056.210439] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9056.212019] ? watchdog+0x40/0x40 [ 9056.213095] ? hrtimer_interrupt+0x128/0x2c0 [ 9056.215041] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9056.216998] ? apic_timer_interrupt+0xf/0x20 [ 9056.218488] [ 9056.219383] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9056.221267] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9056.223432] do_raw_spin_lock+0x177/0x190 [ 9056.224864] _raw_spin_lock+0x21/0x30 [ 9056.226422] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9056.228682] obd_mod_free+0x46/0x140 [ptlrpc] [ 9056.230777] mdc_commit_open+0x91/0xb0 [mdc] [ 9056.232522] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9056.234991] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9056.241550] after_reply+0xe2c/0x1780 [ptlrpc] [ 9056.245979] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9056.252198] ? __next_timer_interrupt+0x160/0x160 [ 9056.260102] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9056.266818] ? do_wait_intr+0xf0/0xf0 [ 9056.269248] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9056.277417] kthread+0x1d1/0x200 [ 9056.285153] ? set_kthread_struct+0x70/0x70 [ 9056.295920] ret_from_fork+0x1f/0x30 [ 9059.810290] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6546s! [ 9059.820476] Showing busy workqueues and worker pools: [ 9059.825899] workqueue events: flags=0x0 [ 9059.828159] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9059.831762] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9059.839149] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9059.844733] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9059.850187] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9059.855276] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9059.862221] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9059.867176] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9059.873612] workqueue events_long: flags=0x0 [ 9059.877461] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9059.881890] pending: lu_objects_flush [obdclass] [ 9059.885468] workqueue events_unbound: flags=0x2 [ 9059.889380] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9059.893821] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9059.898634] workqueue events_power_efficient: flags=0x80 [ 9059.904599] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9059.908681] pending: neigh_periodic_work, do_cache_clean [ 9059.912633] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9059.917669] pending: neigh_periodic_work, check_lifetime [ 9059.923325] workqueue mm_percpu_wq: flags=0x8 [ 9059.926466] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9059.933200] pending: vmstat_update, lru_add_drain_per_cpu [ 9059.937734] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9059.942864] pending: vmstat_update [ 9059.945706] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9059.952856] pending: lru_add_drain_per_cpu BAR(48) [ 9059.956414] workqueue ipv6_addrconf: flags=0x40008 [ 9059.959762] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9059.965880] pending: addrconf_verify_work [ 9059.969484] workqueue knbd0-recv: flags=0x1a [ 9059.971705] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9059.980234] in-flight: 650:recv_work [ 9059.983031] workqueue ptlrpc_pinger: flags=0x2 [ 9059.986249] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9059.991223] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9059.996534] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6546s workers=4 idle: 36 466 29757 [ 9060.001412] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9050s workers=2 idle: 652 [ 9061.132769] rcu: INFO: rcu_sched self-detected stall on CPU [ 9061.138136] rcu: 0-....: (4485600 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 9061.151805] (t=7080192 jiffies g=262441 q=59344) [ 9061.157658] rcu: rcu_sched kthread starved for 6549385 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 9061.181683] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9061.205397] rcu: RCU grace-period kthread stack dump: [ 9061.213969] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9061.223291] Call Trace: [ 9061.224189] __schedule+0x351/0xcb0 [ 9061.230524] schedule+0xc0/0x180 [ 9061.235807] schedule_timeout+0xb4/0x190 [ 9061.237869] ? do_raw_spin_unlock+0x75/0x190 [ 9061.241531] ? __next_timer_interrupt+0x160/0x160 [ 9061.245671] ? prepare_to_swait_event+0xf3/0x140 [ 9061.250206] rcu_gp_kthread+0x8dd/0x15b0 [ 9061.255214] ? force_qs_rnp+0x2c0/0x2c0 [ 9061.257817] kthread+0x1d1/0x200 [ 9061.260005] ? set_kthread_struct+0x70/0x70 [ 9061.262166] ret_from_fork+0x1f/0x30 [ 9061.267584] rcu: Stack dump where RCU GP kthread last ran: [ 9061.272823] Sending NMI from CPU 0 to CPUs 3: [ 9061.280778] NMI backtrace for cpu 3 [ 9061.280784] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9061.280786] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9061.280787] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 9061.280789] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 9061.280791] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 [ 9061.280794] RAX: 0000000000006f19 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9061.280796] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9061.280798] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9061.280799] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9061.280801] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9061.280802] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9061.280803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9061.280804] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9061.280805] Call Trace: [ 9061.280806] [ 9061.280807] ? show_regs.cold.9+0x22/0x2f [ 9061.280808] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9061.280809] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9061.280810] ? nmi_handle+0xbd/0x210 [ 9061.280811] ? default_do_nmi+0x4b/0x1b0 [ 9061.280813] ? do_nmi+0x280/0x380 [ 9061.280813] ? end_repeat_nmi+0x16/0x1f [ 9061.280814] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 9061.280816] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 9061.280817] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 9061.280819] [ 9061.280819] do_raw_spin_lock+0x177/0x190 [ 9061.280820] _raw_spin_lock+0x21/0x30 [ 9061.280822] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9061.280823] ? do_raw_spin_unlock+0x75/0x190 [ 9061.280824] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9061.280825] ? do_wait_intr+0xf0/0xf0 [ 9061.280826] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9061.280827] mdc_reint+0x2d/0x110 [mdc] [ 9061.280828] mdc_unlink+0x658/0x870 [mdc] [ 9061.280829] lmv_unlink+0x34d/0xbd0 [lmv] [ 9061.280830] ll_unlink+0x24d/0x7f0 [lustre] [ 9061.280831] vfs_unlink+0x202/0x330 [ 9061.280832] do_unlinkat+0x2fa/0x460 [ 9061.280833] __x64_sys_unlink+0x37/0x50 [ 9061.280834] do_syscall_64+0xc1/0x440 [ 9061.280835] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9061.280836] RIP: 0033:0x7f83e5f5c10b [ 9061.280837] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9061.280840] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9061.280843] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9061.280844] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9061.280846] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9061.280847] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9061.280849] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9061.283626] NMI backtrace for cpu 0 [ 9061.381576] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9061.384870] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9061.387133] Call Trace: [ 9061.387831] [ 9061.388502] ? dump_stack+0xbb/0x10e [ 9061.389722] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9061.391054] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9061.392248] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9061.393923] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9061.395171] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9061.396516] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9061.397606] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9061.398818] ? update_process_times+0x6d/0xc0 [ 9061.400396] ? tick_sched_handle.isra.18+0x27/0x90 [ 9061.401770] ? tick_sched_timer+0x53/0xb0 [ 9061.402711] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9061.404442] ? tick_sched_handle.isra.18+0x90/0x90 [ 9061.406224] ? hrtimer_interrupt+0x128/0x2c0 [ 9061.407844] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9061.409219] ? apic_timer_interrupt+0xf/0x20 [ 9061.410677] [ 9061.411574] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9061.413659] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9061.415620] ? do_raw_spin_lock+0x177/0x190 [ 9061.417128] ? _raw_spin_lock+0x21/0x30 [ 9061.418146] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9061.420019] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9061.421893] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9061.423546] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9061.424828] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9061.426210] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9061.428005] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9061.429688] ? __next_timer_interrupt+0x160/0x160 [ 9061.430871] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9061.432248] ? do_wait_intr+0xf0/0xf0 [ 9061.433489] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9061.435081] ? kthread+0x1d1/0x200 [ 9061.435783] ? set_kthread_struct+0x70/0x70 [ 9061.436716] ? ret_from_fork+0x1f/0x30 [ 9064.060186] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9064.062932] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9064.104990] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9064.111434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9064.119880] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9064.123843] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9064.130078] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9064.145427] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9064.149090] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9064.156682] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9064.161802] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9064.165714] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9064.169408] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9064.173298] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9064.177172] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9064.180026] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9064.183479] Call Trace: [ 9064.184533] [ 9064.185499] ? show_regs.cold.9+0x22/0x2f [ 9064.188051] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9064.191044] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9064.194983] ? watchdog+0x40/0x40 [ 9064.199485] ? hrtimer_interrupt+0x128/0x2c0 [ 9064.202482] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9064.206615] ? apic_timer_interrupt+0xf/0x20 [ 9064.209064] [ 9064.210250] ? native_safe_halt+0x2e/0x50 [ 9064.211869] kvm_wait+0xcd/0xe0 [ 9064.213911] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9064.216704] do_raw_spin_lock+0x177/0x190 [ 9064.218612] _raw_spin_lock+0x21/0x30 [ 9064.220650] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9064.223533] process_one_work+0x2c8/0x700 [ 9064.225600] worker_thread+0x296/0x6e0 [ 9064.227139] ? rescuer_thread+0x570/0x570 [ 9064.229672] kthread+0x1d1/0x200 [ 9064.231918] ? set_kthread_struct+0x70/0x70 [ 9064.234120] ret_from_fork+0x1f/0x30 [ 9076.059014] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9076.063522] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9076.096876] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9076.105936] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9076.109533] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9076.113744] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9076.128809] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9076.133738] RAX: 0000000000007b85 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9076.139873] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9076.143332] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9076.148191] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9076.154253] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9076.158821] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9076.164945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9076.170597] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9076.173402] Call Trace: [ 9076.175297] [ 9076.176773] ? show_regs.cold.9+0x22/0x2f [ 9076.180689] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9076.182898] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9076.187801] ? watchdog+0x40/0x40 [ 9076.190370] ? hrtimer_interrupt+0x128/0x2c0 [ 9076.191738] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9076.193693] ? apic_timer_interrupt+0xf/0x20 [ 9076.195570] [ 9076.196847] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9076.202443] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9076.204822] do_raw_spin_lock+0x177/0x190 [ 9076.208699] _raw_spin_lock+0x21/0x30 [ 9076.211774] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9076.217384] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9076.222550] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9076.228406] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9076.230340] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9076.232013] ? woken_wake_function+0x30/0x30 [ 9076.233697] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9076.236940] kthread+0x1d1/0x200 [ 9076.240470] ? set_kthread_struct+0x70/0x70 [ 9076.242831] ret_from_fork+0x1f/0x30 [ 9088.056184] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 9088.061184] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9088.062019] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 9088.115510] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9088.118447] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9088.118590] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9088.118596] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9088.118598] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9088.118608] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9088.118611] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9088.118615] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9088.118617] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9088.118618] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9088.118620] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9088.118622] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9088.118624] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9088.118627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9088.118628] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9088.118635] Call Trace: [ 9088.118638] [ 9088.118643] ? show_regs.cold.9+0x22/0x2f [ 9088.118650] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9088.118654] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9088.118657] ? watchdog+0x40/0x40 [ 9088.118660] ? hrtimer_interrupt+0x128/0x2c0 [ 9088.118663] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9088.118667] ? apic_timer_interrupt+0xf/0x20 [ 9088.118671] [ 9088.118673] ? native_safe_halt+0x2e/0x50 [ 9088.118675] kvm_wait+0xcd/0xe0 [ 9088.118680] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9088.118685] do_raw_spin_lock+0x177/0x190 [ 9088.118688] _raw_spin_lock+0x21/0x30 [ 9088.118692] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9088.119125] ? do_raw_spin_unlock+0x75/0x190 [ 9088.119131] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9088.119416] ? do_wait_intr+0xf0/0xf0 [ 9088.119422] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9088.119753] mdc_reint+0x2d/0x110 [mdc] [ 9088.119784] mdc_unlink+0x658/0x870 [mdc] [ 9088.119883] lmv_unlink+0x34d/0xbd0 [lmv] [ 9088.119915] ll_unlink+0x24d/0x7f0 [lustre] [ 9088.120133] vfs_unlink+0x202/0x330 [ 9088.120141] do_unlinkat+0x2fa/0x460 [ 9088.120146] __x64_sys_unlink+0x37/0x50 [ 9088.120149] do_syscall_64+0xc1/0x440 [ 9088.120154] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9088.120160] RIP: 0033:0x7f83e5f5c10b [ 9088.120166] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9088.120333] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9088.120340] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9088.120342] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9088.120344] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9088.120346] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9088.120348] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9088.252967] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9088.257185] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9088.258650] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9088.265601] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9088.269579] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9088.272454] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9088.275582] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9088.281860] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9088.285723] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9088.289072] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9088.292204] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9088.294729] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9088.297533] Call Trace: [ 9088.298417] [ 9088.299084] ? show_regs.cold.9+0x22/0x2f [ 9088.300684] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9088.302404] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9088.304673] ? watchdog+0x40/0x40 [ 9088.306137] ? hrtimer_interrupt+0x128/0x2c0 [ 9088.307621] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9088.309930] ? apic_timer_interrupt+0xf/0x20 [ 9088.311981] [ 9088.312732] ? native_safe_halt+0x2e/0x50 [ 9088.314161] kvm_wait+0xcd/0xe0 [ 9088.315322] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9088.317326] do_raw_spin_lock+0x177/0x190 [ 9088.319453] _raw_spin_lock+0x21/0x30 [ 9088.321784] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9088.325886] obd_mod_free+0x46/0x140 [ptlrpc] [ 9088.328775] mdc_commit_open+0x91/0xb0 [mdc] [ 9088.332578] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9088.336824] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9088.340864] after_reply+0xe2c/0x1780 [ptlrpc] [ 9088.344642] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9088.347802] ? __next_timer_interrupt+0x160/0x160 [ 9088.350761] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9088.353962] ? do_wait_intr+0xf0/0xf0 [ 9088.357135] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9088.361114] kthread+0x1d1/0x200 [ 9088.363400] ? set_kthread_struct+0x70/0x70 [ 9088.365705] ret_from_fork+0x1f/0x30 [ 9090.017039] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6576s! [ 9090.021709] Showing busy workqueues and worker pools: [ 9090.024385] workqueue events: flags=0x0 [ 9090.026414] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9090.030091] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9090.034492] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9090.038217] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9090.042350] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9090.045691] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9090.051313] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9090.055182] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9090.060342] workqueue events_long: flags=0x0 [ 9090.062374] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9090.066815] pending: lu_objects_flush [obdclass] [ 9090.069318] workqueue events_unbound: flags=0x2 [ 9090.071746] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9090.075756] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9090.079210] workqueue events_power_efficient: flags=0x80 [ 9090.082110] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9090.086094] pending: neigh_periodic_work, do_cache_clean [ 9090.089182] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9090.092838] pending: neigh_periodic_work, check_lifetime [ 9090.095486] workqueue mm_percpu_wq: flags=0x8 [ 9090.097370] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9090.100364] pending: vmstat_update, lru_add_drain_per_cpu [ 9090.103193] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9090.105989] pending: vmstat_update [ 9090.107682] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9090.111222] pending: lru_add_drain_per_cpu BAR(48) [ 9090.113376] workqueue ipv6_addrconf: flags=0x40008 [ 9090.115496] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9090.118483] pending: addrconf_verify_work [ 9090.120253] workqueue knbd0-recv: flags=0x1a [ 9090.121861] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9090.126322] in-flight: 650:recv_work [ 9090.128504] workqueue ptlrpc_pinger: flags=0x2 [ 9090.130941] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9090.133584] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9090.136474] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6576s workers=4 idle: 36 466 29757 [ 9090.141083] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9080s workers=2 idle: 652 [ 9092.060466] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9092.065935] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9092.087264] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9092.092242] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9092.095808] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9092.098615] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9092.100596] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9092.108546] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9092.111487] RAX: 000000000000193c RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9092.114462] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9092.117093] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9092.123185] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9092.126969] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9092.130742] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9092.134957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9092.137277] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9092.140388] Call Trace: [ 9092.141608] [ 9092.142494] ? show_regs.cold.9+0x22/0x2f [ 9092.144643] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9092.147096] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9092.149771] ? watchdog+0x40/0x40 [ 9092.151485] ? hrtimer_interrupt+0x128/0x2c0 [ 9092.153973] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9092.156370] ? apic_timer_interrupt+0xf/0x20 [ 9092.158527] [ 9092.159510] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9092.162515] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9092.165521] do_raw_spin_lock+0x177/0x190 [ 9092.167704] _raw_spin_lock+0x21/0x30 [ 9092.169777] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9092.173444] process_one_work+0x2c8/0x700 [ 9092.175908] worker_thread+0x296/0x6e0 [ 9092.177557] ? rescuer_thread+0x570/0x570 [ 9092.179530] kthread+0x1d1/0x200 [ 9092.181171] ? set_kthread_struct+0x70/0x70 [ 9092.183393] ret_from_fork+0x1f/0x30 [ 9104.060040] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9104.071463] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9104.168201] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9104.186225] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9104.197296] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9104.215338] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9104.253796] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9104.268265] RAX: 000000000000078a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9104.276075] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9104.288625] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9104.294861] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9104.303264] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9104.314155] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9104.326145] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9104.330461] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9104.340650] Call Trace: [ 9104.341559] [ 9104.343324] ? show_regs.cold.9+0x22/0x2f [ 9104.346337] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9104.353723] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9104.357919] ? watchdog+0x40/0x40 [ 9104.360134] ? hrtimer_interrupt+0x128/0x2c0 [ 9104.374452] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9104.379235] ? apic_timer_interrupt+0xf/0x20 [ 9104.393554] [ 9104.399727] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9104.404893] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9104.417823] do_raw_spin_lock+0x177/0x190 [ 9104.423734] _raw_spin_lock+0x21/0x30 [ 9104.426051] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9104.440098] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9104.449306] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9104.461888] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9104.472807] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9104.480580] ? woken_wake_function+0x30/0x30 [ 9104.490075] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9104.496367] kthread+0x1d1/0x200 [ 9104.498316] ? set_kthread_struct+0x70/0x70 [ 9104.501039] ret_from_fork+0x1f/0x30 [ 9116.062024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 9116.065017] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9116.066029] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 9116.087846] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9116.087863] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9116.087867] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9116.087882] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9116.087887] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9116.087893] RAX: 0000000000000493 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9116.087896] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9116.087898] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9116.087900] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9116.087903] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9116.087907] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9116.087910] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9116.087912] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9116.087922] Call Trace: [ 9116.087928] [ 9116.087934] ? show_regs.cold.9+0x22/0x2f [ 9116.087944] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9116.087951] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9116.087955] ? watchdog+0x40/0x40 [ 9116.087959] ? hrtimer_interrupt+0x128/0x2c0 [ 9116.087964] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9116.087973] ? apic_timer_interrupt+0xf/0x20 [ 9116.087977] [ 9116.087981] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9116.087985] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9116.087990] do_raw_spin_lock+0x177/0x190 [ 9116.087995] _raw_spin_lock+0x21/0x30 [ 9116.088002] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9116.088286] ? do_raw_spin_unlock+0x75/0x190 [ 9116.088292] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9116.088761] ? do_wait_intr+0xf0/0xf0 [ 9116.088780] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9116.088988] mdc_reint+0x2d/0x110 [mdc] [ 9116.089030] mdc_unlink+0x658/0x870 [mdc] [ 9116.089059] lmv_unlink+0x34d/0xbd0 [lmv] [ 9116.089092] ll_unlink+0x24d/0x7f0 [lustre] [ 9116.089212] vfs_unlink+0x202/0x330 [ 9116.089220] do_unlinkat+0x2fa/0x460 [ 9116.089225] __x64_sys_unlink+0x37/0x50 [ 9116.089229] do_syscall_64+0xc1/0x440 [ 9116.089236] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9116.089241] RIP: 0033:0x7f83e5f5c10b [ 9116.089248] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9116.089252] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9116.089257] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9116.089260] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9116.089262] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9116.089264] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9116.089267] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9116.331614] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9116.361213] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9116.369088] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9116.375640] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9116.380384] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9116.394590] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9116.400206] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9116.405024] RDX: 000000000000030d RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9116.409393] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9116.413519] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9116.419447] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9116.425079] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9116.430646] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9116.435488] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9116.441236] Call Trace: [ 9116.443001] [ 9116.444388] ? show_regs.cold.9+0x22/0x2f [ 9116.447009] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9116.450040] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9116.452345] ? watchdog+0x40/0x40 [ 9116.454689] ? hrtimer_interrupt+0x128/0x2c0 [ 9116.457918] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9116.461209] ? apic_timer_interrupt+0xf/0x20 [ 9116.463339] [ 9116.464349] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9116.467064] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9116.469390] do_raw_spin_lock+0x177/0x190 [ 9116.471350] _raw_spin_lock+0x21/0x30 [ 9116.473989] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9116.477912] obd_mod_free+0x46/0x140 [ptlrpc] [ 9116.481076] mdc_commit_open+0x91/0xb0 [mdc] [ 9116.483550] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9116.487908] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9116.493118] after_reply+0xe2c/0x1780 [ptlrpc] [ 9116.496949] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9116.501442] ? __next_timer_interrupt+0x160/0x160 [ 9116.504241] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9116.509361] ? do_wait_intr+0xf0/0xf0 [ 9116.512278] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9116.516558] kthread+0x1d1/0x200 [ 9116.519226] ? set_kthread_struct+0x70/0x70 [ 9116.521541] ret_from_fork+0x1f/0x30 [ 9120.065462] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9120.079302] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9120.133052] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9120.140160] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9120.150137] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9120.155641] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9120.159513] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9120.191558] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9120.201912] RAX: 0000000000005c8f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9120.208984] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9120.212434] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9120.216232] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9120.220399] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9120.224671] BUG: workqueue lockup - pool [ 9120.233740] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9120.233752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9120.233755] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9120.233763] Call Trace: [ 9120.233769] [ 9120.233777] ? show_regs.cold.9+0x22/0x2f [ 9120.233790] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9120.233796] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9120.233801] ? watchdog+0x40/0x40 [ 9120.233806] ? hrtimer_interrupt+0x128/0x2c0 [ 9120.233810] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9120.233817] ? apic_timer_interrupt+0xf/0x20 [ 9120.233821] [ 9120.233825] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9120.233831] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9120.233839] do_raw_spin_lock+0x177/0x190 [ 9120.233844] _raw_spin_lock+0x21/0x30 [ 9120.233849] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9120.236977] cpus=0-3 flags=0x4 nice=0 stuck for 6607s! [ 9120.237021] Showing busy workqueues and worker pools: [ 9120.237091] workqueue events: flags=0x0 [ 9120.237097] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9120.237107] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9120.237763] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9120.237772] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9120.238148] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9120.238157] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9120.240237] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9120.241000] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9120.241921] workqueue events_long: flags=0x0 [ 9120.241926] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9120.241936] pending: lu_objects_flush [obdclass] [ 9120.242165] workqueue events_unbound: flags=0x2 [ 9120.254906] process_one_work+0x2c8/0x700 [ 9120.254929] worker_thread+0x296/0x6e0 [ 9120.254934] ? rescuer_thread+0x570/0x570 [ 9120.254953] kthread+0x1d1/0x200 [ 9120.254958] ? set_kthread_struct+0x70/0x70 [ 9120.254962] ret_from_fork+0x1f/0x30 [ 9120.420977] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9120.425085] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9120.429402] workqueue events_power_efficient: flags=0x80 [ 9120.433018] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9120.437258] pending: neigh_periodic_work, do_cache_clean [ 9120.442265] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9120.450256] pending: neigh_periodic_work, check_lifetime [ 9120.455138] workqueue mm_percpu_wq: flags=0x8 [ 9120.461317] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9120.468048] pending: vmstat_update, lru_add_drain_per_cpu [ 9120.478654] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9120.492525] pending: vmstat_update [ 9120.497589] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9120.520793] pending: lru_add_drain_per_cpu BAR(48) [ 9120.526279] workqueue ipv6_addrconf: flags=0x40008 [ 9120.544503] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9120.560152] pending: addrconf_verify_work [ 9120.569371] workqueue knbd0-recv: flags=0x1a [ 9120.582904] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9120.596409] in-flight: 650:recv_work [ 9120.602380] workqueue ptlrpc_pinger: flags=0x2 [ 9120.612219] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9120.629133] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9120.641804] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6607s workers=4 idle: 36 466 29757 [ 9120.661713] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9110s workers=2 idle: 652 [ 9132.062133] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9132.068619] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9132.101015] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9132.108044] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9132.117077] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9132.119910] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9132.136525] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9132.144258] RAX: 000000000000510a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9132.147349] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9132.151095] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9132.154905] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9132.167752] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9132.172058] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9132.175414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9132.178253] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9132.181081] Call Trace: [ 9132.182020] [ 9132.183235] ? show_regs.cold.9+0x22/0x2f [ 9132.185495] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9132.189989] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9132.192983] ? watchdog+0x40/0x40 [ 9132.195932] ? hrtimer_interrupt+0x128/0x2c0 [ 9132.201270] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9132.204807] ? apic_timer_interrupt+0xf/0x20 [ 9132.206993] [ 9132.208082] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9132.211050] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9132.216104] do_raw_spin_lock+0x177/0x190 [ 9132.219922] _raw_spin_lock+0x21/0x30 [ 9132.221365] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9132.236161] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9132.244665] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9132.253711] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9132.260462] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9132.264501] ? woken_wake_function+0x30/0x30 [ 9132.271157] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9132.281011] kthread+0x1d1/0x200 [ 9132.286645] ? set_kthread_struct+0x70/0x70 [ 9132.292948] ret_from_fork+0x1f/0x30 [ 9144.062968] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 9144.067269] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9144.101231] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9144.112893] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9144.119239] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9144.122208] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9144.131744] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9144.135467] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9144.138426] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9144.144920] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9144.150287] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9144.159448] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9144.163549] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9144.171732] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9144.177962] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9144.188159] Call Trace: [ 9144.189104] [ 9144.193150] ? show_regs.cold.9+0x22/0x2f [ 9144.198437] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9144.201928] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9144.204611] ? watchdog+0x40/0x40 [ 9144.205926] ? hrtimer_interrupt+0x128/0x2c0 [ 9144.209109] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9144.212630] ? apic_timer_interrupt+0xf/0x20 [ 9144.216121] [ 9144.216990] ? native_safe_halt+0x2e/0x50 [ 9144.219766] kvm_wait+0xcd/0xe0 [ 9144.224676] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9144.227683] do_raw_spin_lock+0x177/0x190 [ 9144.230812] _raw_spin_lock+0x21/0x30 [ 9144.234581] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9144.238284] ? do_raw_spin_unlock+0x75/0x190 [ 9144.242086] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9144.243719] ? do_wait_intr+0xf0/0xf0 [ 9144.248034] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9144.251620] mdc_reint+0x2d/0x110 [mdc] [ 9144.255172] mdc_unlink+0x658/0x870 [mdc] [ 9144.257180] lmv_unlink+0x34d/0xbd0 [lmv] [ 9144.259463] ll_unlink+0x24d/0x7f0 [lustre] [ 9144.261419] vfs_unlink+0x202/0x330 [ 9144.262845] do_unlinkat+0x2fa/0x460 [ 9144.264876] __x64_sys_unlink+0x37/0x50 [ 9144.265834] do_syscall_64+0xc1/0x440 [ 9144.267442] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9144.272082] RIP: 0033:0x7f83e5f5c10b [ 9144.274065] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9144.286568] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9144.291021] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9144.294854] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9144.298144] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9144.302303] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9144.306552] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9148.057961] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 9148.060028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9148.069638] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9148.069719] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9148.069727] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9148.069730] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9148.069745] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9148.069750] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9148.069754] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9148.069756] RDX: 000000000000266a RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9148.069759] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9148.069760] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9148.069763] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9148.069766] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9148.069770] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9148.069772] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9148.069782] Call Trace: [ 9148.069787] [ 9148.069793] ? show_regs.cold.9+0x22/0x2f [ 9148.069802] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9148.069808] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9148.069813] ? watchdog+0x40/0x40 [ 9148.069816] ? hrtimer_interrupt+0x128/0x2c0 [ 9148.069820] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9148.069829] ? apic_timer_interrupt+0xf/0x20 [ 9148.069834] [ 9148.069837] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9148.069842] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9148.069847] do_raw_spin_lock+0x177/0x190 [ 9148.069851] _raw_spin_lock+0x21/0x30 [ 9148.069857] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9148.070127] obd_mod_free+0x46/0x140 [ptlrpc] [ 9148.077372] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9148.077461] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9148.077469] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9148.077476] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9148.077778] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9148.077789] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9148.077794] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9148.077798] RAX: 000000000000689b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9148.077801] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9148.077803] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9148.077805] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9148.077808] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9148.077811] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9148.077814] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9148.077817] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9148.077827] Call Trace: [ 9148.077834] [ 9148.077841] ? show_regs.cold.9+0x22/0x2f [ 9148.077856] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9148.077866] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9148.077871] ? watchdog+0x40/0x40 [ 9148.077876] ? hrtimer_interrupt+0x128/0x2c0 [ 9148.077880] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9148.077897] ? apic_timer_interrupt+0xf/0x20 [ 9148.077906] [ 9148.077909] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9148.077914] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9148.077920] do_raw_spin_lock+0x177/0x190 [ 9148.077924] _raw_spin_lock+0x21/0x30 [ 9148.077929] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9148.078148] process_one_work+0x2c8/0x700 [ 9148.078157] worker_thread+0x296/0x6e0 [ 9148.078162] ? rescuer_thread+0x570/0x570 [ 9148.078166] kthread+0x1d1/0x200 [ 9148.078171] ? set_kthread_struct+0x70/0x70 [ 9148.078174] ret_from_fork+0x1f/0x30 [ 9148.443046] mdc_commit_open+0x91/0xb0 [mdc] [ 9148.446294] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9148.447927] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9148.457271] after_reply+0xe2c/0x1780 [ptlrpc] [ 9148.471486] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9148.473492] ? __next_timer_interrupt+0x160/0x160 [ 9148.479333] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9148.483799] ? do_wait_intr+0xf0/0xf0 [ 9148.485922] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9148.490277] kthread+0x1d1/0x200 [ 9148.496817] ? set_kthread_struct+0x70/0x70 [ 9148.498204] ret_from_fork+0x1f/0x30 [ 9150.944813] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6637s! [ 9150.951512] Showing busy workqueues and worker pools: [ 9150.958343] workqueue events: flags=0x0 [ 9150.963026] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9150.970850] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9150.981984] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9150.987297] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9150.991641] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9150.998754] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9151.002871] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9151.008591] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9151.017985] workqueue events_long: flags=0x0 [ 9151.020622] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9151.030032] pending: lu_objects_flush [obdclass] [ 9151.038932] workqueue events_unbound: flags=0x2 [ 9151.042769] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9151.051263] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9151.058259] workqueue events_power_efficient: flags=0x80 [ 9151.061373] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9151.066749] pending: neigh_periodic_work, do_cache_clean [ 9151.071309] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9151.077057] pending: neigh_periodic_work, check_lifetime [ 9151.085776] workqueue mm_percpu_wq: flags=0x8 [ 9151.089822] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9151.100349] pending: vmstat_update, lru_add_drain_per_cpu [ 9151.106563] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9151.114772] pending: vmstat_update [ 9151.117258] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9151.126917] pending: lru_add_drain_per_cpu BAR(48) [ 9151.133716] workqueue ipv6_addrconf: flags=0x40008 [ 9151.136853] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9151.141652] pending: addrconf_verify_work [ 9151.146918] workqueue knbd0-recv: flags=0x1a [ 9151.149699] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9151.162046] in-flight: 650:recv_work [ 9151.167329] workqueue ptlrpc_pinger: flags=0x2 [ 9151.175639] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9151.177231] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9151.188199] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6638s workers=4 idle: 36 466 29757 [ 9151.195833] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9141s workers=2 idle: 652 [ 9160.059298] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9160.066470] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9160.107411] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9160.123932] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9160.132973] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9160.138427] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9160.152367] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9160.165902] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9160.179271] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9160.195863] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9160.203864] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9160.208763] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9160.211938] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9160.217958] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9160.223150] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9160.233155] Call Trace: [ 9160.234790] [ 9160.236195] ? show_regs.cold.9+0x22/0x2f [ 9160.238264] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9160.241513] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9160.244183] ? watchdog+0x40/0x40 [ 9160.246653] ? hrtimer_interrupt+0x128/0x2c0 [ 9160.251824] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9160.261138] ? apic_timer_interrupt+0xf/0x20 [ 9160.264034] [ 9160.265796] ? native_safe_halt+0x2e/0x50 [ 9160.267277] kvm_wait+0xcd/0xe0 [ 9160.273236] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9160.276241] do_raw_spin_lock+0x177/0x190 [ 9160.279411] _raw_spin_lock+0x21/0x30 [ 9160.281750] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9160.286828] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9160.290169] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9160.292880] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9160.300821] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9160.308416] ? woken_wake_function+0x30/0x30 [ 9160.311969] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9160.331820] kthread+0x1d1/0x200 [ 9160.334231] ? set_kthread_struct+0x70/0x70 [ 9160.338784] ret_from_fork+0x1f/0x30 [ 9172.064144] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 9172.079545] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9172.130685] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9172.148168] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9172.162421] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9172.170136] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9172.199012] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9172.206485] RAX: 0000000000007f85 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9172.219283] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9172.227134] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9172.235323] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9172.239395] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9172.243686] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9172.247158] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9172.250293] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9172.254159] Call Trace: [ 9172.255473] [ 9172.256491] ? show_regs.cold.9+0x22/0x2f [ 9172.258787] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9172.264390] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9172.270166] ? watchdog+0x40/0x40 [ 9172.272589] ? hrtimer_interrupt+0x128/0x2c0 [ 9172.280378] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9172.285783] ? apic_timer_interrupt+0xf/0x20 [ 9172.292235] [ 9172.299233] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9172.302135] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9172.311810] do_raw_spin_lock+0x177/0x190 [ 9172.314638] _raw_spin_lock+0x21/0x30 [ 9172.316547] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9172.322598] ? do_raw_spin_unlock+0x75/0x190 [ 9172.327734] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9172.339519] ? do_wait_intr+0xf0/0xf0 [ 9172.342994] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9172.349930] mdc_reint+0x2d/0x110 [mdc] [ 9172.352370] mdc_unlink+0x658/0x870 [mdc] [ 9172.361159] lmv_unlink+0x34d/0xbd0 [lmv] [ 9172.365931] ll_unlink+0x24d/0x7f0 [lustre] [ 9172.371051] vfs_unlink+0x202/0x330 [ 9172.375921] do_unlinkat+0x2fa/0x460 [ 9172.380509] __x64_sys_unlink+0x37/0x50 [ 9172.385065] do_syscall_64+0xc1/0x440 [ 9172.388169] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9172.392437] RIP: 0033:0x7f83e5f5c10b [ 9172.394715] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9172.413827] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9172.422718] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9172.433767] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9172.439911] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9172.443701] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9172.447921] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9176.057339] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 9176.060031] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9176.069859] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) [ 9176.073800] Modules linked in: [ 9176.073800] obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [ 9176.083782] lustre(O) [ 9176.085051] rpcsec_gss_krb5 [ 9176.088080] osp(O) [ 9176.089267] auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9176.089308] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9176.089315] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9176.089318] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9176.090400] ofd(O) [ 9176.091357] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9176.101589] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9176.101657] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9176.101664] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9176.101670] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9176.108276] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9176.108291] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9176.108293] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9176.108295] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9176.108298] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9176.108300] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9176.108304] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9176.108307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9176.108309] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9176.108318] Call Trace: [ 9176.108326] [ 9176.108332] ? show_regs.cold.9+0x22/0x2f [ 9176.108344] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9176.108351] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9176.108355] ? watchdog+0x40/0x40 [ 9176.108359] ? hrtimer_interrupt+0x128/0x2c0 [ 9176.108363] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9176.108370] ? apic_timer_interrupt+0xf/0x20 [ 9176.108374] [ 9176.108377] ? native_safe_halt+0x2e/0x50 [ 9176.108381] kvm_wait+0xcd/0xe0 [ 9176.108387] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9176.108394] do_raw_spin_lock+0x177/0x190 [ 9176.108398] _raw_spin_lock+0x21/0x30 [ 9176.108404] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9176.108661] obd_mod_free+0x46/0x140 [ptlrpc] [ 9176.108869] mdc_commit_open+0x91/0xb0 [mdc] [ 9176.108907] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9176.109346] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9176.109542] after_reply+0xe2c/0x1780 [ptlrpc] [ 9176.109737] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9176.118408] [ 9176.118418] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9176.118436] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9176.118440] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9176.118446] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9176.118448] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9176.118451] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9176.118453] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9176.118456] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9176.118460] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9176.118463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9176.118465] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9176.118474] Call Trace: [ 9176.118480] [ 9176.118485] ? show_regs.cold.9+0x22/0x2f [ 9176.118495] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9176.118502] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9176.118506] ? watchdog+0x40/0x40 [ 9176.118510] ? hrtimer_interrupt+0x128/0x2c0 [ 9176.118515] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9176.118522] ? apic_timer_interrupt+0xf/0x20 [ 9176.118527] [ 9176.118530] ? native_safe_halt+0x2e/0x50 [ 9176.118533] kvm_wait+0xcd/0xe0 [ 9176.118541] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9176.118551] do_raw_spin_lock+0x177/0x190 [ 9176.118556] _raw_spin_lock+0x21/0x30 [ 9176.118560] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9176.119286] process_one_work+0x2c8/0x700 [ 9176.119295] worker_thread+0x296/0x6e0 [ 9176.119300] ? rescuer_thread+0x570/0x570 [ 9176.119304] kthread+0x1d1/0x200 [ 9176.119309] ? set_kthread_struct+0x70/0x70 [ 9176.119314] ret_from_fork+0x1f/0x30 [ 9176.334788] ? __next_timer_interrupt+0x160/0x160 [ 9176.336868] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9176.339385] ? do_wait_intr+0xf0/0xf0 [ 9176.341069] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9176.343597] kthread+0x1d1/0x200 [ 9176.345299] ? set_kthread_struct+0x70/0x70 [ 9176.346955] ret_from_fork+0x1f/0x30 [ 9181.665028] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6668s! [ 9181.674534] Showing busy workqueues and worker pools: [ 9181.678398] workqueue events: flags=0x0 [ 9181.683030] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9181.688968] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9181.697073] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9181.701315] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9181.706519] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9181.715457] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9181.726068] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9181.733023] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9181.748418] workqueue events_long: flags=0x0 [ 9181.751265] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9181.758823] pending: lu_objects_flush [obdclass] [ 9181.766418] workqueue events_unbound: flags=0x2 [ 9181.773623] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9181.782536] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9181.791361] workqueue events_power_efficient: flags=0x80 [ 9181.799995] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9181.814995] pending: neigh_periodic_work, do_cache_clean [ 9181.824875] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9181.832113] pending: neigh_periodic_work, check_lifetime [ 9181.834488] workqueue mm_percpu_wq: flags=0x8 [ 9181.841861] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9181.848815] pending: vmstat_update, lru_add_drain_per_cpu [ 9181.853977] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9181.857723] pending: vmstat_update [ 9181.859306] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9181.866607] pending: lru_add_drain_per_cpu BAR(48) [ 9181.869806] workqueue ipv6_addrconf: flags=0x40008 [ 9181.878052] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9181.881133] pending: addrconf_verify_work [ 9181.883966] workqueue knbd0-recv: flags=0x1a [ 9181.885948] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9181.889995] in-flight: 650:recv_work [ 9181.892442] workqueue ptlrpc_pinger: flags=0x2 [ 9181.894959] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9181.899361] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9181.905176] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6668s workers=4 idle: 36 466 29757 [ 9181.909631] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9171s workers=2 idle: 652 [ 9188.059023] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 9188.061388] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9188.079661] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9188.083797] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9188.086802] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9188.088638] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9188.095652] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9188.098002] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9188.100570] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9188.103075] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9188.105690] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9188.108425] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9188.111115] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9188.114513] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9188.116613] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9188.119252] Call Trace: [ 9188.120098] [ 9188.120931] ? show_regs.cold.9+0x22/0x2f [ 9188.122543] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9188.124457] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9188.126179] ? watchdog+0x40/0x40 [ 9188.127422] ? hrtimer_interrupt+0x128/0x2c0 [ 9188.129001] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9188.131150] ? apic_timer_interrupt+0xf/0x20 [ 9188.132690] [ 9188.133579] ? native_safe_halt+0x2e/0x50 [ 9188.135255] kvm_wait+0xcd/0xe0 [ 9188.136494] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9188.138519] do_raw_spin_lock+0x177/0x190 [ 9188.140096] _raw_spin_lock+0x21/0x30 [ 9188.141519] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9188.145786] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9188.149220] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9188.152681] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9188.155991] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9188.160362] ? woken_wake_function+0x30/0x30 [ 9188.163168] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9188.166864] kthread+0x1d1/0x200 [ 9188.168659] ? set_kthread_struct+0x70/0x70 [ 9188.171556] ret_from_fork+0x1f/0x30 [ 9200.062053] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 9200.066755] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9200.106201] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9200.112856] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9200.118192] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9200.121739] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9200.133401] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9200.138522] RAX: 0000000000003f90 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9200.158811] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9200.174900] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9200.193447] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9200.201547] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9200.207541] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9200.210576] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9200.221266] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9200.234081] Call Trace: [ 9200.235781] [ 9200.240728] ? show_regs.cold.9+0x22/0x2f [ 9200.245409] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9200.255527] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9200.259105] ? watchdog+0x40/0x40 [ 9200.264569] ? hrtimer_interrupt+0x128/0x2c0 [ 9200.270238] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9200.278829] ? apic_timer_interrupt+0xf/0x20 [ 9200.287066] [ 9200.291102] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9200.298289] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9200.300167] do_raw_spin_lock+0x177/0x190 [ 9200.314492] _raw_spin_lock+0x21/0x30 [ 9200.320098] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9200.330019] ? do_raw_spin_unlock+0x75/0x190 [ 9200.333521] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9200.339685] ? do_wait_intr+0xf0/0xf0 [ 9200.347415] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9200.355581] mdc_reint+0x2d/0x110 [mdc] [ 9200.361703] mdc_unlink+0x658/0x870 [mdc] [ 9200.364484] lmv_unlink+0x34d/0xbd0 [lmv] [ 9200.367440] ll_unlink+0x24d/0x7f0 [lustre] [ 9200.370501] vfs_unlink+0x202/0x330 [ 9200.372805] do_unlinkat+0x2fa/0x460 [ 9200.377349] __x64_sys_unlink+0x37/0x50 [ 9200.381362] do_syscall_64+0xc1/0x440 [ 9200.382659] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9200.396100] RIP: 0033:0x7f83e5f5c10b [ 9200.407194] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9200.437483] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9200.444775] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9200.450129] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9200.454670] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9200.463756] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9200.475699] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9204.060051] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9204.063363] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9204.144604] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9204.162601] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9204.177341] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9204.187762] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9204.195108] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9204.214561] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9204.218723] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9204.227292] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9204.234800] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9204.244331] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9204.252492] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9204.259556] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9204.265624] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9204.273500] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9204.284255] Call Trace: [ 9204.287233] [ 9204.293787] ? show_regs.cold.9+0x22/0x2f [ 9204.296708] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9204.309382] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9204.313644] ? watchdog+0x40/0x40 [ 9204.319018] ? hrtimer_interrupt+0x128/0x2c0 [ 9204.322126] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9204.335616] ? apic_timer_interrupt+0xf/0x20 [ 9204.337211] [ 9204.345820] ? native_safe_halt+0x2e/0x50 [ 9204.354847] kvm_wait+0xcd/0xe0 [ 9204.365457] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9204.367304] do_raw_spin_lock+0x177/0x190 [ 9204.374700] _raw_spin_lock+0x21/0x30 [ 9204.381718] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9204.385255] process_one_work+0x2c8/0x700 [ 9204.386606] worker_thread+0x296/0x6e0 [ 9204.389819] ? rescuer_thread+0x570/0x570 [ 9204.391489] kthread+0x1d1/0x200 [ 9204.394068] ? set_kthread_struct+0x70/0x70 [ 9204.395824] ret_from_fork+0x1f/0x30 [ 9208.062764] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 9208.067364] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9208.097323] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9208.103031] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9208.105511] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9208.108043] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9208.115657] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9208.123095] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9208.125316] RDX: 0000000000007387 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9208.134124] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9208.138292] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9208.141363] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9208.151529] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9208.156417] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9208.158934] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9208.161489] Call Trace: [ 9208.162469] [ 9208.163329] ? show_regs.cold.9+0x22/0x2f [ 9208.164627] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9208.166352] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9208.168119] ? watchdog+0x40/0x40 [ 9208.169415] ? hrtimer_interrupt+0x128/0x2c0 [ 9208.171370] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9208.174571] ? apic_timer_interrupt+0xf/0x20 [ 9208.176369] [ 9208.177640] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9208.179732] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9208.182518] do_raw_spin_lock+0x177/0x190 [ 9208.184216] _raw_spin_lock+0x21/0x30 [ 9208.185423] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9208.187855] obd_mod_free+0x46/0x140 [ptlrpc] [ 9208.190030] mdc_commit_open+0x91/0xb0 [mdc] [ 9208.191600] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9208.193162] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9208.195273] after_reply+0xe2c/0x1780 [ptlrpc] [ 9208.198818] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9208.200746] ? __next_timer_interrupt+0x160/0x160 [ 9208.203404] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9208.205270] ? do_wait_intr+0xf0/0xf0 [ 9208.206975] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9208.209089] kthread+0x1d1/0x200 [ 9208.210245] ? set_kthread_struct+0x70/0x70 [ 9208.211666] ret_from_fork+0x1f/0x30 [ 9212.388676] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6699s! [ 9212.402486] Showing busy workqueues and worker pools: [ 9212.409894] workqueue events: flags=0x0 [ 9212.411996] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9212.415976] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9212.421987] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9212.425421] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9212.428061] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9212.431553] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9212.435265] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9212.438331] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9212.442266] workqueue events_long: flags=0x0 [ 9212.444224] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9212.447018] pending: lu_objects_flush [obdclass] [ 9212.449578] workqueue events_unbound: flags=0x2 [ 9212.452102] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9212.456567] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9212.466628] workqueue events_power_efficient: flags=0x80 [ 9212.477089] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9212.485821] pending: neigh_periodic_work, do_cache_clean [ 9212.490947] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9212.494742] pending: neigh_periodic_work, check_lifetime [ 9212.506059] workqueue mm_percpu_wq: flags=0x8 [ 9212.514853] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9212.529995] pending: vmstat_update, lru_add_drain_per_cpu [ 9212.540011] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9212.549084] pending: vmstat_update [ 9212.554529] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9212.562513] pending: lru_add_drain_per_cpu BAR(48) [ 9212.572859] workqueue ipv6_addrconf: flags=0x40008 [ 9212.577362] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9212.587232] pending: addrconf_verify_work [ 9212.594600] workqueue knbd0-recv: flags=0x1a [ 9212.599868] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9212.611296] in-flight: 650:recv_work [ 9212.614069] workqueue ptlrpc_pinger: flags=0x2 [ 9212.617020] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9212.618307] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9212.623745] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6699s workers=4 idle: 36 466 29757 [ 9212.635117] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9202s workers=2 idle: 652 [ 9216.059416] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 9216.062530] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9216.100088] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9216.107500] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9216.116064] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9216.119299] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9216.136333] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9216.141331] RAX: 000000000000744b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9216.144441] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9216.147272] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9216.151692] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9216.157357] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9216.164404] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9216.172660] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9216.183585] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9216.198361] Call Trace: [ 9216.200246] [ 9216.201129] ? show_regs.cold.9+0x22/0x2f [ 9216.206485] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9216.214362] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9216.223218] ? watchdog+0x40/0x40 [ 9216.227242] ? hrtimer_interrupt+0x128/0x2c0 [ 9216.229201] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9216.231902] ? apic_timer_interrupt+0xf/0x20 [ 9216.238266] [ 9216.239832] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9216.243694] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9216.251237] do_raw_spin_lock+0x177/0x190 [ 9216.256605] _raw_spin_lock+0x21/0x30 [ 9216.261318] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9216.271934] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9216.281124] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9216.283901] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9216.287401] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9216.294468] ? woken_wake_function+0x30/0x30 [ 9216.300187] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9216.307458] kthread+0x1d1/0x200 [ 9216.312121] ? set_kthread_struct+0x70/0x70 [ 9216.320373] ret_from_fork+0x1f/0x30 [ 9228.065126] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 9228.068369] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9228.097062] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9228.101752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9228.108194] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9228.111173] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9228.123521] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9228.130419] RAX: 0000000000003c78 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9228.134725] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9228.139849] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9228.141786] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9228.151211] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9228.153168] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9228.159267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9228.161289] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9228.168270] Call Trace: [ 9228.169123] [ 9228.169880] ? show_regs.cold.9+0x22/0x2f [ 9228.171600] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9228.177618] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9228.182193] ? watchdog+0x40/0x40 [ 9228.184685] ? hrtimer_interrupt+0x128/0x2c0 [ 9228.187012] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9228.188448] ? apic_timer_interrupt+0xf/0x20 [ 9228.193344] [ 9228.194110] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9228.196243] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9228.198337] do_raw_spin_lock+0x177/0x190 [ 9228.200462] _raw_spin_lock+0x21/0x30 [ 9228.204004] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9228.206918] ? do_raw_spin_unlock+0x75/0x190 [ 9228.209301] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9228.212056] ? do_wait_intr+0xf0/0xf0 [ 9228.213166] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9228.217153] mdc_reint+0x2d/0x110 [mdc] [ 9228.220479] mdc_unlink+0x658/0x870 [mdc] [ 9228.223754] lmv_unlink+0x34d/0xbd0 [lmv] [ 9228.226982] ll_unlink+0x24d/0x7f0 [lustre] [ 9228.229266] vfs_unlink+0x202/0x330 [ 9228.237951] do_unlinkat+0x2fa/0x460 [ 9228.238947] __x64_sys_unlink+0x37/0x50 [ 9228.240032] do_syscall_64+0xc1/0x440 [ 9228.245151] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9228.249022] RIP: 0033:0x7f83e5f5c10b [ 9228.255395] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9228.270995] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9228.276488] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9228.283245] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9228.288025] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9228.294189] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9228.303195] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9232.060072] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 9232.062954] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9232.104573] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9232.115241] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9232.120472] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9232.127945] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9232.133459] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9232.153168] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9232.157900] RAX: 000000000000790e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9232.165931] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9232.173962] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9232.183516] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9232.191015] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9232.202468] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9232.215057] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9232.221856] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9232.225075] Call Trace: [ 9232.227218] [ 9232.228156] ? show_regs.cold.9+0x22/0x2f [ 9232.230586] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9232.235206] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9232.240042] ? watchdog+0x40/0x40 [ 9232.241908] ? hrtimer_interrupt+0x128/0x2c0 [ 9232.244177] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9232.246275] ? apic_timer_interrupt+0xf/0x20 [ 9232.254004] [ 9232.254814] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9232.260438] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9232.264230] do_raw_spin_lock+0x177/0x190 [ 9232.267815] _raw_spin_lock+0x21/0x30 [ 9232.269576] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9232.277571] process_one_work+0x2c8/0x700 [ 9232.282851] worker_thread+0x296/0x6e0 [ 9232.285502] ? rescuer_thread+0x570/0x570 [ 9232.291981] kthread+0x1d1/0x200 [ 9232.295200] ? set_kthread_struct+0x70/0x70 [ 9232.298084] ret_from_fork+0x1f/0x30 [ 9240.056031] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 9240.059409] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9240.081269] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9240.085984] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9240.089814] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9240.092453] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9240.101155] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9240.104980] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9240.108367] RDX: 00000000000003bd RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9240.112498] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9240.117274] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9240.121454] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9240.125395] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9240.129204] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9240.132389] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9240.137006] Call Trace: [ 9240.138720] [ 9240.140659] ? show_regs.cold.9+0x22/0x2f [ 9240.144336] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9240.147008] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9240.149726] ? watchdog+0x40/0x40 [ 9240.151730] ? hrtimer_interrupt+0x128/0x2c0 [ 9240.154371] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9240.157388] ? apic_timer_interrupt+0xf/0x20 [ 9240.159674] [ 9240.161310] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9240.164263] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9240.167500] do_raw_spin_lock+0x177/0x190 [ 9240.169940] _raw_spin_lock+0x21/0x30 [ 9240.172152] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9240.177392] obd_mod_free+0x46/0x140 [ptlrpc] [ 9240.181637] mdc_commit_open+0x91/0xb0 [mdc] [ 9240.183112] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9240.188585] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9240.194884] after_reply+0xe2c/0x1780 [ptlrpc] [ 9240.199137] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9240.204679] ? __next_timer_interrupt+0x160/0x160 [ 9240.210906] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9240.215879] ? do_wait_intr+0xf0/0xf0 [ 9240.221111] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9240.229072] kthread+0x1d1/0x200 [ 9240.232492] ? set_kthread_struct+0x70/0x70 [ 9240.240982] ret_from_fork+0x1f/0x30 [ 9241.137714] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 9241.140833] rcu: 0-....: (4582860 ticks this GP) idle=30a/1/0x4000000000000000 softirq=128519/128519 fqs=126395 [ 9241.146793] (detected by 1, t=7260178 jiffies, g=262441, q=59420) [ 9241.150326] Sending NMI from CPU 1 to CPUs 0: [ 9241.153439] NMI backtrace for cpu 0 [ 9241.153445] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9241.153448] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9241.153452] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9241.153455] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9241.153460] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 [ 9241.153462] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9241.153464] RDX: 0000000000007d9c RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9241.153466] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9241.153467] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9241.153469] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9241.153470] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9241.153472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9241.153473] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9241.153475] Call Trace: [ 9241.153475] [ 9241.153476] ? show_regs.cold.9+0x22/0x2f [ 9241.153477] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9241.153478] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9241.153479] ? nmi_handle+0xbd/0x210 [ 9241.153483] ? default_do_nmi+0x4b/0x1b0 [ 9241.153484] ? do_nmi+0x280/0x380 [ 9241.153485] ? end_repeat_nmi+0x16/0x1f [ 9241.153486] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9241.153487] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9241.153488] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9241.153489] [ 9241.153490] do_raw_spin_lock+0x177/0x190 [ 9241.153491] _raw_spin_lock+0x21/0x30 [ 9241.153494] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9241.153495] obd_mod_free+0x46/0x140 [ptlrpc] [ 9241.153496] mdc_commit_open+0x91/0xb0 [mdc] [ 9241.153497] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9241.153498] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9241.153501] after_reply+0xe2c/0x1780 [ptlrpc] [ 9241.153502] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9241.153503] ? __next_timer_interrupt+0x160/0x160 [ 9241.153504] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9241.153505] ? do_wait_intr+0xf0/0xf0 [ 9241.153506] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9241.153507] kthread+0x1d1/0x200 [ 9241.153508] ? set_kthread_struct+0x70/0x70 [ 9241.153509] ret_from_fork+0x1f/0x30 [ 9241.154013] rcu: rcu_sched kthread starved for 6729365 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 9241.250021] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9241.254286] rcu: RCU grace-period kthread stack dump: [ 9241.256938] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9241.260938] Call Trace: [ 9241.262430] __schedule+0x351/0xcb0 [ 9241.263911] schedule+0xc0/0x180 [ 9241.265177] schedule_timeout+0xb4/0x190 [ 9241.267358] ? do_raw_spin_unlock+0x75/0x190 [ 9241.269755] ? __next_timer_interrupt+0x160/0x160 [ 9241.271249] ? prepare_to_swait_event+0xf3/0x140 [ 9241.272698] rcu_gp_kthread+0x8dd/0x15b0 [ 9241.274730] ? force_qs_rnp+0x2c0/0x2c0 [ 9241.276203] kthread+0x1d1/0x200 [ 9241.277949] ? set_kthread_struct+0x70/0x70 [ 9241.279359] ret_from_fork+0x1f/0x30 [ 9241.280421] rcu: Stack dump where RCU GP kthread last ran: [ 9241.283304] NMI backtrace for cpu 1 [ 9241.284654] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9241.289473] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9241.292978] Call Trace: [ 9241.293921] [ 9241.294985] ? dump_stack+0xbb/0x10e [ 9241.297290] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9241.299734] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9241.301973] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9241.303935] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9241.306151] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9241.308869] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 9241.313183] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [ 9241.315796] ? update_process_times+0x6d/0xc0 [ 9241.317757] ? tick_sched_handle.isra.18+0x27/0x90 [ 9241.320296] ? tick_sched_timer+0x53/0xb0 [ 9241.323236] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9241.325361] ? tick_sched_handle.isra.18+0x90/0x90 [ 9241.327121] ? hrtimer_interrupt+0x128/0x2c0 [ 9241.329116] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9241.331271] ? apic_timer_interrupt+0xf/0x20 [ 9241.333404] [ 9241.334364] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9241.336939] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9241.339679] ? do_raw_spin_lock+0x177/0x190 [ 9241.341786] ? _raw_spin_lock+0x21/0x30 [ 9241.343656] ? ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9241.347382] ? ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9241.350502] ? ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9241.353475] ? ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9241.356248] ? ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9241.359216] ? woken_wake_function+0x30/0x30 [ 9241.361611] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9241.365266] ? kthread+0x1d1/0x200 [ 9241.368241] ? set_kthread_struct+0x70/0x70 [ 9241.370385] ? ret_from_fork+0x1f/0x30 [ 9243.104130] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6729s! [ 9243.107942] Showing busy workqueues and worker pools: [ 9243.110122] workqueue events: flags=0x0 [ 9243.112112] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9243.115255] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9243.119264] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9243.122429] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9243.125675] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9243.128794] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9243.132715] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9243.135913] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9243.140239] workqueue events_long: flags=0x0 [ 9243.142127] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9243.145630] pending: lu_objects_flush [obdclass] [ 9243.148077] workqueue events_unbound: flags=0x2 [ 9243.150139] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9243.153201] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9243.156131] workqueue events_power_efficient: flags=0x80 [ 9243.158865] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9243.162107] pending: neigh_periodic_work, do_cache_clean [ 9243.164900] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9243.167894] pending: neigh_periodic_work, check_lifetime [ 9243.171014] workqueue mm_percpu_wq: flags=0x8 [ 9243.173788] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9243.176877] pending: vmstat_update, lru_add_drain_per_cpu [ 9243.179963] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9243.183076] pending: vmstat_update [ 9243.184795] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9243.188005] pending: lru_add_drain_per_cpu BAR(48) [ 9243.190475] workqueue ipv6_addrconf: flags=0x40008 [ 9243.192188] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9243.195090] pending: addrconf_verify_work [ 9243.196472] workqueue knbd0-recv: flags=0x1a [ 9243.197979] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9243.200137] in-flight: 650:recv_work [ 9243.201612] workqueue ptlrpc_pinger: flags=0x2 [ 9243.203012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9243.205119] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9243.207759] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6730s workers=4 idle: 36 466 29757 [ 9243.210370] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9233s workers=2 idle: 652 [ 9256.063114] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 9256.072301] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9256.142752] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9256.174351] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9256.192426] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9256.211685] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9256.256957] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9256.282932] RAX: 0000000000000b92 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9256.295477] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9256.317191] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9256.332756] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9256.349472] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9256.365898] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9256.381740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9256.385836] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9256.403900] Call Trace: [ 9256.408335] [ 9256.413343] ? show_regs.cold.9+0x22/0x2f [ 9256.426510] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9256.437234] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9256.439021] ? watchdog+0x40/0x40 [ 9256.450781] ? hrtimer_interrupt+0x128/0x2c0 [ 9256.456695] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9256.469941] ? apic_timer_interrupt+0xf/0x20 [ 9256.477878] [ 9256.478615] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9256.486468] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9256.496883] do_raw_spin_lock+0x177/0x190 [ 9256.506574] _raw_spin_lock+0x21/0x30 [ 9256.507843] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9256.520797] ? do_raw_spin_unlock+0x75/0x190 [ 9256.531983] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9256.535984] ? do_wait_intr+0xf0/0xf0 [ 9256.541884] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9256.553792] mdc_reint+0x2d/0x110 [mdc] [ 9256.557847] mdc_unlink+0x658/0x870 [mdc] [ 9256.562822] lmv_unlink+0x34d/0xbd0 [lmv] [ 9256.568888] ll_unlink+0x24d/0x7f0 [lustre] [ 9256.574859] vfs_unlink+0x202/0x330 [ 9256.580575] do_unlinkat+0x2fa/0x460 [ 9256.587540] __x64_sys_unlink+0x37/0x50 [ 9256.593023] do_syscall_64+0xc1/0x440 [ 9256.597921] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9256.605036] RIP: 0033:0x7f83e5f5c10b [ 9256.606445] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9256.626195] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9256.635399] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9256.647801] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9256.657207] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9256.661749] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9256.665936] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9260.060013] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 9260.063552] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9260.083887] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9260.089007] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9260.093700] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9260.096826] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9260.099731] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9260.109272] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9260.113179] RAX: 000000000000171a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9260.117470] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9260.121061] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9260.123979] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9260.127430] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9260.130929] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9260.135623] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9260.138501] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9260.142809] Call Trace: [ 9260.144166] [ 9260.145496] ? show_regs.cold.9+0x22/0x2f [ 9260.147761] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9260.150244] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9260.151748] ? watchdog+0x40/0x40 [ 9260.153293] ? hrtimer_interrupt+0x128/0x2c0 [ 9260.155763] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9260.158434] ? apic_timer_interrupt+0xf/0x20 [ 9260.160548] [ 9260.161682] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9260.164744] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9260.168363] do_raw_spin_lock+0x177/0x190 [ 9260.170678] _raw_spin_lock+0x21/0x30 [ 9260.172661] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9260.176826] process_one_work+0x2c8/0x700 [ 9260.180620] worker_thread+0x296/0x6e0 [ 9260.186152] ? rescuer_thread+0x570/0x570 [ 9260.189643] kthread+0x1d1/0x200 [ 9260.192934] ? set_kthread_struct+0x70/0x70 [ 9260.198215] ret_from_fork+0x1f/0x30 [ 9268.056073] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 9268.059012] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9268.059492] Modules linked in: [ 9268.063049] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9268.063180] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9268.063187] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9268.063190] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9268.063203] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9268.063207] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9268.063212] RAX: 0000000000002247 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9268.063215] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9268.063217] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9268.063219] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9268.063222] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9268.063226] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9268.063229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9268.063231] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9268.063241] Call Trace: [ 9268.063246] [ 9268.063333] ? show_regs.cold.9+0x22/0x2f [ 9268.063343] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9268.063350] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9268.063357] ? watchdog+0x40/0x40 [ 9268.063361] ? hrtimer_interrupt+0x128/0x2c0 [ 9268.063365] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9268.063373] ? apic_timer_interrupt+0xf/0x20 [ 9268.063377] [ 9268.063383] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9268.063387] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9268.063393] do_raw_spin_lock+0x177/0x190 [ 9268.063398] _raw_spin_lock+0x21/0x30 [ 9268.063402] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9268.063698] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9268.106720] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9268.106798] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9268.106806] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9268.106809] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9268.106821] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9268.106825] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9268.106830] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9268.106832] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9268.106835] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9268.106838] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9268.106840] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9268.106844] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9268.106847] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9268.106849] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9268.106858] Call Trace: [ 9268.106864] [ 9268.106869] ? show_regs.cold.9+0x22/0x2f [ 9268.106878] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9268.106885] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9268.106890] ? watchdog+0x40/0x40 [ 9268.106894] ? hrtimer_interrupt+0x128/0x2c0 [ 9268.106899] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9268.106906] ? apic_timer_interrupt+0xf/0x20 [ 9268.106911] [ 9268.106914] ? native_safe_halt+0x2e/0x50 [ 9268.106918] kvm_wait+0xcd/0xe0 [ 9268.106925] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9268.106933] do_raw_spin_lock+0x177/0x190 [ 9268.106937] _raw_spin_lock+0x21/0x30 [ 9268.106943] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9268.120661] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9268.128096] obd_mod_free+0x46/0x140 [ptlrpc] [ 9268.133629] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9268.134151] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9268.146158] mdc_commit_open+0x91/0xb0 [mdc] [ 9268.153421] ? woken_wake_function+0x30/0x30 [ 9268.153448] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9268.153822] kthread+0x1d1/0x200 [ 9268.153828] ? set_kthread_struct+0x70/0x70 [ 9268.153833] ret_from_fork+0x1f/0x30 [ 9268.393674] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9268.398213] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9268.401623] after_reply+0xe2c/0x1780 [ptlrpc] [ 9268.404871] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9268.409464] ? __next_timer_interrupt+0x160/0x160 [ 9268.412160] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9268.415128] ? do_wait_intr+0xf0/0xf0 [ 9268.417174] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9268.420754] kthread+0x1d1/0x200 [ 9268.422835] ? set_kthread_struct+0x70/0x70 [ 9268.425743] ret_from_fork+0x1f/0x30 [ 9273.321206] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6760s! [ 9273.326680] Showing busy workqueues and worker pools: [ 9273.329395] workqueue events: flags=0x0 [ 9273.331420] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9273.337816] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9273.348636] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9273.356041] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9273.372556] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9273.378783] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9273.389535] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9273.393677] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9273.401456] workqueue events_long: flags=0x0 [ 9273.404022] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9273.408054] pending: lu_objects_flush [obdclass] [ 9273.411524] workqueue events_unbound: flags=0x2 [ 9273.414267] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9273.418588] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9273.426518] workqueue events_power_efficient: flags=0x80 [ 9273.438633] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9273.451158] pending: neigh_periodic_work, do_cache_clean [ 9273.457903] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9273.465507] pending: neigh_periodic_work, check_lifetime [ 9273.468835] workqueue mm_percpu_wq: flags=0x8 [ 9273.480336] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9273.490028] pending: vmstat_update, lru_add_drain_per_cpu [ 9273.492288] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9273.503758] pending: vmstat_update [ 9273.513685] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9273.523727] pending: lru_add_drain_per_cpu BAR(48) [ 9273.532837] workqueue ipv6_addrconf: flags=0x40008 [ 9273.540357] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9273.542616] pending: addrconf_verify_work [ 9273.555987] workqueue knbd0-recv: flags=0x1a [ 9273.561020] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9273.566899] in-flight: 650:recv_work [ 9273.570525] workqueue ptlrpc_pinger: flags=0x2 [ 9273.574520] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9273.579674] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9273.585794] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6760s workers=4 idle: 36 466 29757 [ 9273.590928] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9263s workers=2 idle: 652 [ 9284.062079] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 9284.067591] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9284.089392] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9284.095455] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9284.100084] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9284.103698] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9284.110978] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 9284.114203] RAX: 0000000000005286 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9284.117463] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9284.119295] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9284.122257] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9284.123525] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9284.127136] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9284.132717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9284.136236] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9284.139584] Call Trace: [ 9284.140501] [ 9284.141367] ? show_regs.cold.9+0x22/0x2f [ 9284.143542] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9284.145755] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9284.148159] ? watchdog+0x40/0x40 [ 9284.149428] ? hrtimer_interrupt+0x128/0x2c0 [ 9284.151564] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9284.153647] ? apic_timer_interrupt+0xf/0x20 [ 9284.155763] [ 9284.156676] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9284.159131] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9284.161714] do_raw_spin_lock+0x177/0x190 [ 9284.163689] _raw_spin_lock+0x21/0x30 [ 9284.165596] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9284.169263] ? do_raw_spin_unlock+0x75/0x190 [ 9284.171274] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9284.174215] ? do_wait_intr+0xf0/0xf0 [ 9284.176086] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9284.178805] mdc_reint+0x2d/0x110 [mdc] [ 9284.180584] mdc_unlink+0x658/0x870 [mdc] [ 9284.182360] lmv_unlink+0x34d/0xbd0 [lmv] [ 9284.184333] ll_unlink+0x24d/0x7f0 [lustre] [ 9284.186648] vfs_unlink+0x202/0x330 [ 9284.188416] do_unlinkat+0x2fa/0x460 [ 9284.190150] __x64_sys_unlink+0x37/0x50 [ 9284.191830] do_syscall_64+0xc1/0x440 [ 9284.193631] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9284.196165] RIP: 0033:0x7f83e5f5c10b [ 9284.197599] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9284.207046] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9284.211270] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9284.214749] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9284.218357] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9284.221868] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9284.225602] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9288.061302] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 9288.075476] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9288.113689] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9288.120234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9288.123653] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9288.126571] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9288.128596] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9288.135645] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9288.139104] RAX: 0000000000007616 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9288.141162] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9288.143452] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9288.145690] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9288.148411] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9288.150941] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9288.153971] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9288.155816] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9288.158216] Call Trace: [ 9288.159378] [ 9288.160299] ? show_regs.cold.9+0x22/0x2f [ 9288.162039] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9288.163801] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9288.165370] ? watchdog+0x40/0x40 [ 9288.166607] ? hrtimer_interrupt+0x128/0x2c0 [ 9288.168993] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9288.170823] ? apic_timer_interrupt+0xf/0x20 [ 9288.172415] [ 9288.173290] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9288.175406] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9288.177454] do_raw_spin_lock+0x177/0x190 [ 9288.178963] _raw_spin_lock+0x21/0x30 [ 9288.180284] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9288.182460] process_one_work+0x2c8/0x700 [ 9288.184041] worker_thread+0x296/0x6e0 [ 9288.185495] ? rescuer_thread+0x570/0x570 [ 9288.187835] kthread+0x1d1/0x200 [ 9288.189214] ? set_kthread_struct+0x70/0x70 [ 9288.190495] ret_from_fork+0x1f/0x30 [ 9296.060942] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9296.070667] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9296.125873] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9296.139153] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9296.143584] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9296.146968] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9296.163079] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9296.168360] RAX: 0000000000007f84 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9296.172323] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9296.178406] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9296.186859] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9296.192114] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9296.196034] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9296.201512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9296.209508] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9296.214234] Call Trace: [ 9296.220936] [ 9296.221631] ? show_regs.cold.9+0x22/0x2f [ 9296.223135] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9296.231311] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9296.236205] ? watchdog+0x40/0x40 [ 9296.238555] ? hrtimer_interrupt+0x128/0x2c0 [ 9296.242608] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9296.244713] ? apic_timer_interrupt+0xf/0x20 [ 9296.252235] [ 9296.253755] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9296.260903] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9296.265755] do_raw_spin_lock+0x177/0x190 [ 9296.276315] _raw_spin_lock+0x21/0x30 [ 9296.279525] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9296.285549] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9296.288566] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9296.292033] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9296.298401] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9296.301467] ? woken_wake_function+0x30/0x30 [ 9296.305227] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9296.308492] kthread+0x1d1/0x200 [ 9296.310189] ? set_kthread_struct+0x70/0x70 [ 9296.315408] ret_from_fork+0x1f/0x30 [ 9300.058752] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 9300.068251] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9300.113824] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9300.126564] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9300.132662] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9300.139547] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9300.157468] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9300.165412] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9300.173327] RDX: 0000000000003371 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9300.184865] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9300.194371] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9300.203524] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9300.213287] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9300.223521] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9300.232398] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9300.250408] Call Trace: [ 9300.257818] [ 9300.261602] ? show_regs.cold.9+0x22/0x2f [ 9300.270512] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9300.279678] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9300.288350] ? watchdog+0x40/0x40 [ 9300.293267] ? hrtimer_interrupt+0x128/0x2c0 [ 9300.298373] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9300.304924] ? apic_timer_interrupt+0xf/0x20 [ 9300.306595] [ 9300.309577] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9300.316931] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9300.329593] do_raw_spin_lock+0x177/0x190 [ 9300.340978] _raw_spin_lock+0x21/0x30 [ 9300.342167] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9300.355872] obd_mod_free+0x46/0x140 [ptlrpc] [ 9300.367157] mdc_commit_open+0x91/0xb0 [mdc] [ 9300.372560] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9300.381145] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9300.389138] after_reply+0xe2c/0x1780 [ptlrpc] [ 9300.394709] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9300.403040] ? __next_timer_interrupt+0x160/0x160 [ 9300.407617] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9300.418128] ? do_wait_intr+0xf0/0xf0 [ 9300.425475] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9300.437732] kthread+0x1d1/0x200 [ 9300.445017] ? set_kthread_struct+0x70/0x70 [ 9300.448609] ret_from_fork+0x1f/0x30 [ 9304.038508] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6790s! [ 9304.043355] Showing busy workqueues and worker pools: [ 9304.046201] workqueue events: flags=0x0 [ 9304.053435] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9304.061366] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9304.071788] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9304.076207] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9304.080364] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9304.083784] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9304.093350] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9304.102604] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9304.109029] workqueue events_long: flags=0x0 [ 9304.111401] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9304.114940] pending: lu_objects_flush [obdclass] [ 9304.118338] workqueue events_unbound: flags=0x2 [ 9304.122114] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9304.129217] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9304.132669] workqueue events_power_efficient: flags=0x80 [ 9304.136037] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9304.140074] pending: neigh_periodic_work, do_cache_clean [ 9304.144103] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9304.147922] pending: neigh_periodic_work, check_lifetime [ 9304.150762] workqueue mm_percpu_wq: flags=0x8 [ 9304.153317] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9304.159312] pending: vmstat_update, lru_add_drain_per_cpu [ 9304.163037] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9304.167319] pending: vmstat_update [ 9304.169463] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9304.176073] pending: lru_add_drain_per_cpu BAR(48) [ 9304.181126] workqueue ipv6_addrconf: flags=0x40008 [ 9304.184580] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9304.191364] pending: addrconf_verify_work [ 9304.197856] workqueue knbd0-recv: flags=0x1a [ 9304.201039] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9304.204718] in-flight: 650:recv_work [ 9304.206679] workqueue ptlrpc_pinger: flags=0x2 [ 9304.209503] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9304.213720] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9304.218192] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6791s workers=4 idle: 36 466 29757 [ 9304.223156] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9294s workers=2 idle: 652 [ 9312.063905] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 9312.071606] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9312.105038] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9312.113298] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9312.121218] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9312.124674] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9312.136768] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9312.143099] RAX: 0000000000002003 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9312.147163] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9312.151714] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9312.157849] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9312.162977] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9312.168645] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9312.176713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9312.181423] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9312.185156] Call Trace: [ 9312.186576] [ 9312.187692] ? show_regs.cold.9+0x22/0x2f [ 9312.193345] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9312.198582] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9312.201753] ? watchdog+0x40/0x40 [ 9312.206519] ? hrtimer_interrupt+0x128/0x2c0 [ 9312.208799] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9312.215417] ? apic_timer_interrupt+0xf/0x20 [ 9312.221253] [ 9312.222308] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9312.226201] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9312.233105] do_raw_spin_lock+0x177/0x190 [ 9312.237510] _raw_spin_lock+0x21/0x30 [ 9312.239266] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9312.245674] ? do_raw_spin_unlock+0x75/0x190 [ 9312.249096] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9312.253356] ? do_wait_intr+0xf0/0xf0 [ 9312.256808] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9312.263638] mdc_reint+0x2d/0x110 [mdc] [ 9312.266220] mdc_unlink+0x658/0x870 [mdc] [ 9312.271758] lmv_unlink+0x34d/0xbd0 [lmv] [ 9312.274949] ll_unlink+0x24d/0x7f0 [lustre] [ 9312.277195] vfs_unlink+0x202/0x330 [ 9312.282824] do_unlinkat+0x2fa/0x460 [ 9312.285809] __x64_sys_unlink+0x37/0x50 [ 9312.288231] do_syscall_64+0xc1/0x440 [ 9312.292897] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9312.297769] RIP: 0033:0x7f83e5f5c10b [ 9312.300389] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9312.316368] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9312.326461] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9312.330659] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9312.333775] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9312.337778] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9312.342583] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9316.061487] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 9316.066106] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9316.092031] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9316.097132] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9316.101775] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9316.104691] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9316.107558] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9316.116177] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9316.119713] RAX: 0000000000003caf RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9316.122534] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9316.125688] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9316.128382] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9316.130452] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9316.133160] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9316.136904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9316.139532] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9316.146224] Call Trace: [ 9316.147677] [ 9316.149213] ? show_regs.cold.9+0x22/0x2f [ 9316.151180] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9316.154618] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9316.156774] ? watchdog+0x40/0x40 [ 9316.160454] ? hrtimer_interrupt+0x128/0x2c0 [ 9316.164351] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9316.168612] ? apic_timer_interrupt+0xf/0x20 [ 9316.177400] [ 9316.178529] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9316.182417] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9316.185498] do_raw_spin_lock+0x177/0x190 [ 9316.194366] _raw_spin_lock+0x21/0x30 [ 9316.199485] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9316.207646] process_one_work+0x2c8/0x700 [ 9316.212666] worker_thread+0x296/0x6e0 [ 9316.215747] ? rescuer_thread+0x570/0x570 [ 9316.219223] kthread+0x1d1/0x200 [ 9316.221300] ? set_kthread_struct+0x70/0x70 [ 9316.224046] ret_from_fork+0x1f/0x30 [ 9324.064235] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9324.071066] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9324.108089] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9324.119997] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9324.126068] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9324.129432] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9324.137405] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9324.141235] RAX: 000000000000428a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9324.146541] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9324.151604] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9324.154947] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9324.159922] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9324.162925] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9324.166245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9324.169003] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9324.171893] Call Trace: [ 9324.173625] [ 9324.175024] ? show_regs.cold.9+0x22/0x2f [ 9324.177163] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9324.183343] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9324.185925] ? watchdog+0x40/0x40 [ 9324.187684] ? hrtimer_interrupt+0x128/0x2c0 [ 9324.189520] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9324.192495] ? apic_timer_interrupt+0xf/0x20 [ 9324.194836] [ 9324.196012] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9324.198466] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9324.201172] do_raw_spin_lock+0x177/0x190 [ 9324.207339] _raw_spin_lock+0x21/0x30 [ 9324.208565] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9324.213078] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9324.220503] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9324.223336] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9324.226217] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9324.230992] ? woken_wake_function+0x30/0x30 [ 9324.232761] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9324.235993] kthread+0x1d1/0x200 [ 9324.237990] ? set_kthread_struct+0x70/0x70 [ 9324.249035] ret_from_fork+0x1f/0x30 [ 9332.056051] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 9332.066615] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9332.109953] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9332.120294] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9332.125110] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9332.132453] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9332.159340] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9332.175334] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9332.180187] RDX: 000000000000766f RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9332.187007] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9332.199204] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9332.207179] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9332.212948] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9332.216897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9332.221222] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9332.224936] Call Trace: [ 9332.226202] [ 9332.228371] ? show_regs.cold.9+0x22/0x2f [ 9332.231299] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9332.236227] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9332.237762] ? watchdog+0x40/0x40 [ 9332.241175] ? hrtimer_interrupt+0x128/0x2c0 [ 9332.243818] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9332.246356] ? apic_timer_interrupt+0xf/0x20 [ 9332.248608] [ 9332.252817] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9332.255586] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9332.258889] do_raw_spin_lock+0x177/0x190 [ 9332.260395] _raw_spin_lock+0x21/0x30 [ 9332.264940] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9332.269312] obd_mod_free+0x46/0x140 [ptlrpc] [ 9332.272888] mdc_commit_open+0x91/0xb0 [mdc] [ 9332.275949] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9332.279559] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9332.281777] after_reply+0xe2c/0x1780 [ptlrpc] [ 9332.289207] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9332.291837] ? __next_timer_interrupt+0x160/0x160 [ 9332.294285] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9332.296383] ? do_wait_intr+0xf0/0xf0 [ 9332.298017] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9332.299795] kthread+0x1d1/0x200 [ 9332.301130] ? set_kthread_struct+0x70/0x70 [ 9332.302872] ret_from_fork+0x1f/0x30 [ 9334.240854] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6821s! [ 9334.245749] Showing busy workqueues and worker pools: [ 9334.248935] workqueue events: flags=0x0 [ 9334.254218] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9334.256590] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9334.263090] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9334.269919] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9334.279567] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9334.293904] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9334.305709] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9334.312900] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9334.319737] workqueue events_long: flags=0x0 [ 9334.327102] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9334.331756] pending: lu_objects_flush [obdclass] [ 9334.337190] workqueue events_unbound: flags=0x2 [ 9334.339948] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9334.345825] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9334.349300] workqueue events_power_efficient: flags=0x80 [ 9334.354662] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9334.356950] pending: neigh_periodic_work, do_cache_clean [ 9334.366844] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9334.373049] pending: neigh_periodic_work, check_lifetime [ 9334.376206] workqueue mm_percpu_wq: flags=0x8 [ 9334.378203] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9334.383254] pending: vmstat_update, lru_add_drain_per_cpu [ 9334.390765] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9334.400024] pending: vmstat_update [ 9334.404034] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9334.409188] pending: lru_add_drain_per_cpu BAR(48) [ 9334.413588] workqueue ipv6_addrconf: flags=0x40008 [ 9334.421018] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9334.427430] pending: addrconf_verify_work [ 9334.429762] workqueue knbd0-recv: flags=0x1a [ 9334.431926] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9334.440943] in-flight: 650:recv_work [ 9334.442249] workqueue ptlrpc_pinger: flags=0x2 [ 9334.447185] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9334.451440] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9334.455825] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6821s workers=4 idle: 36 466 29757 [ 9334.459294] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9324s workers=2 idle: 652 [ 9340.062026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 9340.065141] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9340.084724] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9340.098840] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9340.106088] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9340.111586] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9340.120769] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9340.124296] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9340.127134] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9340.130297] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9340.134015] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9340.137980] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9340.141766] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9340.145935] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9340.148520] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9340.150749] Call Trace: [ 9340.151964] [ 9340.152766] ? show_regs.cold.9+0x22/0x2f [ 9340.153868] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9340.155392] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9340.157020] ? watchdog+0x40/0x40 [ 9340.158087] ? hrtimer_interrupt+0x128/0x2c0 [ 9340.160017] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9340.162299] ? apic_timer_interrupt+0xf/0x20 [ 9340.164331] [ 9340.165199] ? native_safe_halt+0x2e/0x50 [ 9340.167035] kvm_wait+0xcd/0xe0 [ 9340.168459] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9340.170545] do_raw_spin_lock+0x177/0x190 [ 9340.172150] _raw_spin_lock+0x21/0x30 [ 9340.173840] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9340.176066] ? do_raw_spin_unlock+0x75/0x190 [ 9340.177939] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9340.180247] ? do_wait_intr+0xf0/0xf0 [ 9340.181827] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9340.184342] mdc_reint+0x2d/0x110 [mdc] [ 9340.185896] mdc_unlink+0x658/0x870 [mdc] [ 9340.187744] lmv_unlink+0x34d/0xbd0 [lmv] [ 9340.189224] ll_unlink+0x24d/0x7f0 [lustre] [ 9340.191150] vfs_unlink+0x202/0x330 [ 9340.192719] do_unlinkat+0x2fa/0x460 [ 9340.194205] __x64_sys_unlink+0x37/0x50 [ 9340.195875] do_syscall_64+0xc1/0x440 [ 9340.197686] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9340.199798] RIP: 0033:0x7f83e5f5c10b [ 9340.201262] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9340.208944] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9340.212316] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9340.216041] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9340.219670] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9340.222772] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9340.225829] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9344.060107] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 9344.064443] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9344.081995] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9344.085541] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9344.088285] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9344.090506] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9344.092703] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9344.099769] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9344.102779] RAX: 0000000000001f7e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9344.105664] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9344.107908] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9344.109997] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9344.112259] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9344.114706] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9344.117102] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9344.118818] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9344.121300] Call Trace: [ 9344.122164] [ 9344.122803] ? show_regs.cold.9+0x22/0x2f [ 9344.124115] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9344.125813] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9344.127750] ? watchdog+0x40/0x40 [ 9344.128935] ? hrtimer_interrupt+0x128/0x2c0 [ 9344.130397] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9344.132357] ? apic_timer_interrupt+0xf/0x20 [ 9344.133777] [ 9344.134830] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9344.136926] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9344.139261] do_raw_spin_lock+0x177/0x190 [ 9344.140719] _raw_spin_lock+0x21/0x30 [ 9344.142304] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9344.144862] process_one_work+0x2c8/0x700 [ 9344.146381] worker_thread+0x296/0x6e0 [ 9344.147929] ? rescuer_thread+0x570/0x570 [ 9344.149635] kthread+0x1d1/0x200 [ 9344.150829] ? set_kthread_struct+0x70/0x70 [ 9344.152308] ret_from_fork+0x1f/0x30 [ 9352.059267] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9352.067308] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9352.113658] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9352.127906] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9352.136688] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9352.146118] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9352.161099] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9352.177985] RAX: 0000000000006999 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9352.186064] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9352.195118] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9352.204253] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9352.214212] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9352.221760] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9352.234930] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9352.244424] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9352.259328] Call Trace: [ 9352.263029] [ 9352.264119] ? show_regs.cold.9+0x22/0x2f [ 9352.267160] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9352.269713] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9352.276336] ? watchdog+0x40/0x40 [ 9352.284826] ? hrtimer_interrupt+0x128/0x2c0 [ 9352.289529] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9352.291333] ? apic_timer_interrupt+0xf/0x20 [ 9352.300041] [ 9352.302308] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9352.305591] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9352.317092] do_raw_spin_lock+0x177/0x190 [ 9352.319210] _raw_spin_lock+0x21/0x30 [ 9352.323373] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9352.328655] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9352.331336] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9352.334014] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9352.336835] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9352.339970] ? woken_wake_function+0x30/0x30 [ 9352.342052] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9352.345577] kthread+0x1d1/0x200 [ 9352.347634] ? set_kthread_struct+0x70/0x70 [ 9352.357792] ret_from_fork+0x1f/0x30 [ 9360.056028] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 9360.063165] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9360.109438] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9360.116768] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9360.121063] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9360.123185] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9360.132498] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9360.137220] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9360.143545] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9360.148349] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9360.152832] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9360.155959] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9360.159260] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9360.163735] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9360.167714] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9360.171119] Call Trace: [ 9360.173900] [ 9360.174841] ? show_regs.cold.9+0x22/0x2f [ 9360.177037] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9360.180562] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9360.184861] ? watchdog+0x40/0x40 [ 9360.187123] ? hrtimer_interrupt+0x128/0x2c0 [ 9360.189985] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9360.193904] ? apic_timer_interrupt+0xf/0x20 [ 9360.195935] [ 9360.197779] ? native_safe_halt+0x2e/0x50 [ 9360.199965] kvm_wait+0xcd/0xe0 [ 9360.201887] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9360.204119] do_raw_spin_lock+0x177/0x190 [ 9360.206474] _raw_spin_lock+0x21/0x30 [ 9360.210370] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9360.213299] obd_mod_free+0x46/0x140 [ptlrpc] [ 9360.218011] mdc_commit_open+0x91/0xb0 [mdc] [ 9360.221602] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9360.225048] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9360.227691] after_reply+0xe2c/0x1780 [ptlrpc] [ 9360.230256] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9360.234515] ? __next_timer_interrupt+0x160/0x160 [ 9360.237601] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9360.244212] ? do_wait_intr+0xf0/0xf0 [ 9360.247970] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9360.256220] kthread+0x1d1/0x200 [ 9360.257952] ? set_kthread_struct+0x70/0x70 [ 9360.260382] ret_from_fork+0x1f/0x30 [ 9364.960066] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6851s! [ 9364.964578] Showing busy workqueues and worker pools: [ 9364.969745] workqueue events: flags=0x0 [ 9364.972590] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9364.978791] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9364.990286] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9364.996466] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9365.006055] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9365.012580] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9365.025809] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9365.036583] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9365.058718] workqueue events_long: flags=0x0 [ 9365.062736] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9365.080798] pending: lu_objects_flush [obdclass] [ 9365.093763] workqueue events_unbound: flags=0x2 [ 9365.101251] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9365.106434] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9365.118141] workqueue events_power_efficient: flags=0x80 [ 9365.124441] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9365.131598] pending: neigh_periodic_work, do_cache_clean [ 9365.141535] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9365.154051] pending: neigh_periodic_work, check_lifetime [ 9365.173871] workqueue mm_percpu_wq: flags=0x8 [ 9365.180100] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9365.191012] pending: vmstat_update, lru_add_drain_per_cpu [ 9365.207490] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9365.216400] pending: vmstat_update [ 9365.217906] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9365.243622] pending: lru_add_drain_per_cpu BAR(48) [ 9365.250072] workqueue ipv6_addrconf: flags=0x40008 [ 9365.258572] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9365.271256] pending: addrconf_verify_work [ 9365.278547] workqueue knbd0-recv: flags=0x1a [ 9365.280435] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9365.288198] in-flight: 650:recv_work [ 9365.295606] workqueue ptlrpc_pinger: flags=0x2 [ 9365.300850] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9365.312026] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9365.322775] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6852s workers=4 idle: 36 466 29757 [ 9365.339386] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9355s workers=2 idle: 652 [ 9368.062045] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 9368.069574] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9368.108592] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9368.118215] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9368.123159] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9368.126619] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9368.139148] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9368.147398] RAX: 00000000000066e0 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9368.155786] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9368.159641] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9368.163177] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9368.168055] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9368.174437] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9368.179641] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9368.183648] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9368.189757] Call Trace: [ 9368.197814] [ 9368.198790] ? show_regs.cold.9+0x22/0x2f [ 9368.201606] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9368.207025] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9368.210522] ? watchdog+0x40/0x40 [ 9368.212427] ? hrtimer_interrupt+0x128/0x2c0 [ 9368.217853] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9368.220925] ? apic_timer_interrupt+0xf/0x20 [ 9368.222802] [ 9368.225810] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9368.229044] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9368.232145] do_raw_spin_lock+0x177/0x190 [ 9368.234628] _raw_spin_lock+0x21/0x30 [ 9368.236666] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9368.240169] ? do_raw_spin_unlock+0x75/0x190 [ 9368.243813] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9368.248193] ? do_wait_intr+0xf0/0xf0 [ 9368.250247] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9368.254958] mdc_reint+0x2d/0x110 [mdc] [ 9368.257219] mdc_unlink+0x658/0x870 [mdc] [ 9368.263168] lmv_unlink+0x34d/0xbd0 [lmv] [ 9368.266991] ll_unlink+0x24d/0x7f0 [lustre] [ 9368.269557] vfs_unlink+0x202/0x330 [ 9368.272154] do_unlinkat+0x2fa/0x460 [ 9368.274210] __x64_sys_unlink+0x37/0x50 [ 9368.276597] do_syscall_64+0xc1/0x440 [ 9368.278848] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9368.282470] RIP: 0033:0x7f83e5f5c10b [ 9368.285757] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9368.302542] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9368.308223] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9368.312597] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9368.317933] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9368.333930] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9368.347667] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9372.061391] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9372.069583] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9372.131387] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9372.148051] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9372.157712] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9372.167017] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9372.169577] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9372.179802] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9372.189640] RAX: 0000000000006c91 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9372.196649] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9372.206114] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9372.212347] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9372.216033] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9372.220513] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9372.231005] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9372.234958] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9372.243557] Call Trace: [ 9372.247714] [ 9372.250077] ? show_regs.cold.9+0x22/0x2f [ 9372.252173] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9372.259600] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9372.260861] ? watchdog+0x40/0x40 [ 9372.261794] ? hrtimer_interrupt+0x128/0x2c0 [ 9372.273515] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9372.281516] ? apic_timer_interrupt+0xf/0x20 [ 9372.287161] [ 9372.290510] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9372.297579] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9372.299114] do_raw_spin_lock+0x177/0x190 [ 9372.307733] _raw_spin_lock+0x21/0x30 [ 9372.309141] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9372.311212] process_one_work+0x2c8/0x700 [ 9372.312770] worker_thread+0x296/0x6e0 [ 9372.313946] ? rescuer_thread+0x570/0x570 [ 9372.315158] kthread+0x1d1/0x200 [ 9372.316481] ? set_kthread_struct+0x70/0x70 [ 9372.317820] ret_from_fork+0x1f/0x30 [ 9380.059674] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9380.061949] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9380.093473] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9380.101759] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9380.108058] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9380.111021] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9380.123286] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9380.127883] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9380.132304] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9380.135327] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9380.138818] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9380.142117] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9380.144938] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9380.148394] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9380.151489] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9380.154777] Call Trace: [ 9380.155917] [ 9380.156944] ? show_regs.cold.9+0x22/0x2f [ 9380.159000] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9380.160879] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9380.162976] ? watchdog+0x40/0x40 [ 9380.164811] ? hrtimer_interrupt+0x128/0x2c0 [ 9380.166942] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9380.169617] ? apic_timer_interrupt+0xf/0x20 [ 9380.172004] [ 9380.173097] ? native_safe_halt+0x2e/0x50 [ 9380.175026] kvm_wait+0xcd/0xe0 [ 9380.176440] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9380.179157] do_raw_spin_lock+0x177/0x190 [ 9380.181637] _raw_spin_lock+0x21/0x30 [ 9380.183500] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9380.186736] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9380.189894] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9380.192602] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9380.195814] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9380.199295] ? woken_wake_function+0x30/0x30 [ 9380.201749] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9380.205816] kthread+0x1d1/0x200 [ 9380.207479] ? set_kthread_struct+0x70/0x70 [ 9380.209584] ret_from_fork+0x1f/0x30 [ 9392.056008] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 9392.068741] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9392.099432] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9392.106409] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9392.110770] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9392.114229] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9392.120714] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9392.123769] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9392.126897] RDX: 00000000000051cb RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9392.135616] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9392.145509] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9392.153592] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9392.158462] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9392.176960] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9392.186835] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9392.193271] Call Trace: [ 9392.197814] [ 9392.199989] ? show_regs.cold.9+0x22/0x2f [ 9392.208614] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9392.214403] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9392.217909] ? watchdog+0x40/0x40 [ 9392.223201] ? hrtimer_interrupt+0x128/0x2c0 [ 9392.229605] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9392.234297] ? apic_timer_interrupt+0xf/0x20 [ 9392.237619] [ 9392.238596] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9392.245995] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9392.255663] do_raw_spin_lock+0x177/0x190 [ 9392.260812] _raw_spin_lock+0x21/0x30 [ 9392.264400] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9392.271510] obd_mod_free+0x46/0x140 [ptlrpc] [ 9392.278993] mdc_commit_open+0x91/0xb0 [mdc] [ 9392.281862] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9392.295200] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9392.303236] after_reply+0xe2c/0x1780 [ptlrpc] [ 9392.307151] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9392.314926] ? __next_timer_interrupt+0x160/0x160 [ 9392.323609] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9392.326624] ? do_wait_intr+0xf0/0xf0 [ 9392.328712] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9392.331810] kthread+0x1d1/0x200 [ 9392.333105] ? set_kthread_struct+0x70/0x70 [ 9392.340359] ret_from_fork+0x1f/0x30 [ 9395.681226] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6882s! [ 9395.685570] Showing busy workqueues and worker pools: [ 9395.688828] workqueue events: flags=0x0 [ 9395.690991] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9395.693600] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9395.706380] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9395.718221] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9395.723165] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9395.725944] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9395.730873] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9395.734306] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9395.741439] workqueue events_long: flags=0x0 [ 9395.744026] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9395.747932] pending: lu_objects_flush [obdclass] [ 9395.751867] workqueue events_unbound: flags=0x2 [ 9395.754252] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9395.758429] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9395.761331] workqueue events_power_efficient: flags=0x80 [ 9395.764306] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9395.767468] pending: neigh_periodic_work, do_cache_clean [ 9395.769942] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9395.773526] pending: neigh_periodic_work, check_lifetime [ 9395.776877] workqueue mm_percpu_wq: flags=0x8 [ 9395.779226] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9395.782806] pending: vmstat_update, lru_add_drain_per_cpu [ 9395.786311] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9395.789519] pending: vmstat_update [ 9395.791291] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9395.794492] pending: lru_add_drain_per_cpu BAR(48) [ 9395.797430] workqueue ipv6_addrconf: flags=0x40008 [ 9395.804643] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9395.812219] pending: addrconf_verify_work [ 9395.814980] workqueue knbd0-recv: flags=0x1a [ 9395.822675] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9395.830063] in-flight: 650:recv_work [ 9395.832522] workqueue ptlrpc_pinger: flags=0x2 [ 9395.836901] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9395.840658] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9395.844417] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6882s workers=4 idle: 36 466 29757 [ 9395.849225] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9385s workers=2 idle: 652 [ 9396.062023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 9396.065328] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9396.092734] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9396.098390] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9396.102262] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9396.104112] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9396.111729] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9396.114924] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9396.118399] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9396.122698] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9396.126514] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9396.130247] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9396.134069] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9396.138890] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9396.142650] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9396.146624] Call Trace: [ 9396.148242] [ 9396.149466] ? show_regs.cold.9+0x22/0x2f [ 9396.151336] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9396.154324] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9396.157088] ? watchdog+0x40/0x40 [ 9396.159743] ? hrtimer_interrupt+0x128/0x2c0 [ 9396.162187] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9396.165003] ? apic_timer_interrupt+0xf/0x20 [ 9396.167162] [ 9396.169004] ? native_safe_halt+0x2e/0x50 [ 9396.170908] kvm_wait+0xcd/0xe0 [ 9396.172778] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9396.175703] do_raw_spin_lock+0x177/0x190 [ 9396.177601] _raw_spin_lock+0x21/0x30 [ 9396.179531] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9396.181513] ? do_raw_spin_unlock+0x75/0x190 [ 9396.183692] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9396.186407] ? do_wait_intr+0xf0/0xf0 [ 9396.188395] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9396.191302] mdc_reint+0x2d/0x110 [mdc] [ 9396.194147] mdc_unlink+0x658/0x870 [mdc] [ 9396.195991] lmv_unlink+0x34d/0xbd0 [lmv] [ 9396.197966] ll_unlink+0x24d/0x7f0 [lustre] [ 9396.200384] vfs_unlink+0x202/0x330 [ 9396.201979] do_unlinkat+0x2fa/0x460 [ 9396.203788] __x64_sys_unlink+0x37/0x50 [ 9396.205913] do_syscall_64+0xc1/0x440 [ 9396.207671] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9396.209819] RIP: 0033:0x7f83e5f5c10b [ 9396.211541] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9396.219835] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9396.222665] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9396.226005] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9396.229337] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9396.233687] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9396.237456] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9400.060096] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9400.064070] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9400.090862] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9400.096120] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9400.101302] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9400.104834] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9400.108760] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9400.123048] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9400.127832] RAX: 0000000000005206 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9400.131837] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9400.135779] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9400.139855] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9400.143803] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9400.148187] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9400.152532] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9400.155896] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9400.160155] Call Trace: [ 9400.161820] [ 9400.163055] ? show_regs.cold.9+0x22/0x2f [ 9400.165573] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9400.167954] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9400.170764] ? watchdog+0x40/0x40 [ 9400.172218] ? hrtimer_interrupt+0x128/0x2c0 [ 9400.174477] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9400.176804] ? apic_timer_interrupt+0xf/0x20 [ 9400.178831] [ 9400.180016] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9400.182870] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9400.185586] do_raw_spin_lock+0x177/0x190 [ 9400.187892] _raw_spin_lock+0x21/0x30 [ 9400.189559] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9400.192243] process_one_work+0x2c8/0x700 [ 9400.194854] worker_thread+0x296/0x6e0 [ 9400.196681] ? rescuer_thread+0x570/0x570 [ 9400.198672] kthread+0x1d1/0x200 [ 9400.200289] ? set_kthread_struct+0x70/0x70 [ 9400.202300] ret_from_fork+0x1f/0x30 [ 9408.061159] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9408.068055] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9408.122507] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9408.135782] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9408.142988] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9408.147653] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9408.160063] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9408.165904] RAX: 000000000000790a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9408.174140] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9408.186778] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9408.196450] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9408.200104] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9408.209833] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9408.220097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9408.227157] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9408.234667] Call Trace: [ 9408.236632] [ 9408.239272] ? show_regs.cold.9+0x22/0x2f [ 9408.242000] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9408.248169] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9408.252790] ? watchdog+0x40/0x40 [ 9408.255795] ? hrtimer_interrupt+0x128/0x2c0 [ 9408.260774] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9408.262462] ? apic_timer_interrupt+0xf/0x20 [ 9408.266130] [ 9408.268072] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9408.272069] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9408.278492] do_raw_spin_lock+0x177/0x190 [ 9408.281513] _raw_spin_lock+0x21/0x30 [ 9408.286121] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9408.293308] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9408.299670] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9408.306080] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9408.308855] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9408.321114] ? woken_wake_function+0x30/0x30 [ 9408.329438] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9408.341824] kthread+0x1d1/0x200 [ 9408.343133] ? set_kthread_struct+0x70/0x70 [ 9408.354811] ret_from_fork+0x1f/0x30 [ 9420.056451] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 9420.060927] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9420.080520] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9420.085463] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9420.089573] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9420.092682] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9420.101044] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9420.104373] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9420.107065] RDX: 0000000000001616 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9420.110406] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9420.113261] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9420.116386] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9420.119427] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9420.122773] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9420.124991] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9420.128059] Call Trace: [ 9420.129177] [ 9420.130896] ? show_regs.cold.9+0x22/0x2f [ 9420.132735] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9420.135553] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9420.138171] ? watchdog+0x40/0x40 [ 9420.140283] ? hrtimer_interrupt+0x128/0x2c0 [ 9420.143117] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9420.145428] ? apic_timer_interrupt+0xf/0x20 [ 9420.147606] [ 9420.148591] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9420.150773] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9420.152911] do_raw_spin_lock+0x177/0x190 [ 9420.154497] _raw_spin_lock+0x21/0x30 [ 9420.156293] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9420.159482] obd_mod_free+0x46/0x140 [ptlrpc] [ 9420.163103] mdc_commit_open+0x91/0xb0 [mdc] [ 9420.164697] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9420.167294] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9420.170001] after_reply+0xe2c/0x1780 [ptlrpc] [ 9420.172307] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9420.174833] ? __next_timer_interrupt+0x160/0x160 [ 9420.176822] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9420.178744] ? do_wait_intr+0xf0/0xf0 [ 9420.180243] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9420.182555] kthread+0x1d1/0x200 [ 9420.183952] ? set_kthread_struct+0x70/0x70 [ 9420.185633] ret_from_fork+0x1f/0x30 [ 9421.142056] rcu: INFO: rcu_sched self-detected stall on CPU [ 9421.144376] rcu: 0-....: (4687907 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 9421.148337] (t=7440187 jiffies g=262441 q=59466) [ 9421.149540] rcu: rcu_sched kthread starved for 6909374 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 9421.152720] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9421.156403] rcu: RCU grace-period kthread stack dump: [ 9421.158240] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9421.162196] Call Trace: [ 9421.163308] __schedule+0x351/0xcb0 [ 9421.164506] schedule+0xc0/0x180 [ 9421.166176] schedule_timeout+0xb4/0x190 [ 9421.167831] ? do_raw_spin_unlock+0x75/0x190 [ 9421.169914] ? __next_timer_interrupt+0x160/0x160 [ 9421.172023] ? prepare_to_swait_event+0xf3/0x140 [ 9421.174041] rcu_gp_kthread+0x8dd/0x15b0 [ 9421.175537] ? force_qs_rnp+0x2c0/0x2c0 [ 9421.177343] kthread+0x1d1/0x200 [ 9421.178649] ? set_kthread_struct+0x70/0x70 [ 9421.180867] ret_from_fork+0x1f/0x30 [ 9421.182411] rcu: Stack dump where RCU GP kthread last ran: [ 9421.184092] Sending NMI from CPU 0 to CPUs 1: [ 9421.191392] NMI backtrace for cpu 1 [ 9421.191403] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9421.191405] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9421.191406] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9421.191408] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9421.191411] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 [ 9421.191414] RAX: 0000000000006d5f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9421.191418] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9421.191420] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9421.191421] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9421.191423] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9421.191424] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9421.191426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9421.191427] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9421.191428] Call Trace: [ 9421.191438] [ 9421.191440] ? show_regs.cold.9+0x22/0x2f [ 9421.191441] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9421.191442] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9421.191443] ? nmi_handle+0xbd/0x210 [ 9421.191444] ? default_do_nmi+0x4b/0x1b0 [ 9421.191445] ? do_nmi+0x280/0x380 [ 9421.191446] ? end_repeat_nmi+0x16/0x1f [ 9421.191447] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9421.191448] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9421.191450] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9421.191451] [ 9421.191451] do_raw_spin_lock+0x177/0x190 [ 9421.191452] _raw_spin_lock+0x21/0x30 [ 9421.191453] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9421.191454] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9421.191456] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9421.191457] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9421.191458] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9421.191462] ? woken_wake_function+0x30/0x30 [ 9421.191464] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9421.191465] kthread+0x1d1/0x200 [ 9421.191466] ? set_kthread_struct+0x70/0x70 [ 9421.191467] ret_from_fork+0x1f/0x30 [ 9421.191556] NMI backtrace for cpu 0 [ 9421.311834] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9421.321110] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9421.332377] Call Trace: [ 9421.334717] [ 9421.335416] ? dump_stack+0xbb/0x10e [ 9421.337458] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9421.339916] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9421.342098] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9421.354917] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9421.363716] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9421.376104] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9421.377679] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9421.386278] ? update_process_times+0x6d/0xc0 [ 9421.393497] ? tick_sched_handle.isra.18+0x27/0x90 [ 9421.395029] ? tick_sched_timer+0x53/0xb0 [ 9421.396377] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9421.409016] ? tick_sched_handle.isra.18+0x90/0x90 [ 9421.413738] ? hrtimer_interrupt+0x128/0x2c0 [ 9421.417603] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9421.422779] ? apic_timer_interrupt+0xf/0x20 [ 9421.428158] [ 9421.436925] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9421.439723] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9421.449511] ? do_raw_spin_lock+0x177/0x190 [ 9421.459050] ? _raw_spin_lock+0x21/0x30 [ 9421.464190] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9421.469513] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9421.473707] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9421.478629] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9421.488291] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9421.490573] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9421.497454] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9421.504023] ? __next_timer_interrupt+0x160/0x160 [ 9421.507887] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9421.510347] ? do_wait_intr+0xf0/0xf0 [ 9421.514220] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9421.517756] ? kthread+0x1d1/0x200 [ 9421.522799] ? set_kthread_struct+0x70/0x70 [ 9421.525153] ? ret_from_fork+0x1f/0x30 [ 9424.062020] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 9424.065485] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9424.086388] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9424.091872] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9424.096175] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9424.098257] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9424.106945] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9424.110309] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9424.113806] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9424.117790] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9424.121114] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9424.124664] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9424.136097] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9424.143361] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9424.149922] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9424.157951] Call Trace: [ 9424.158796] [ 9424.159485] ? show_regs.cold.9+0x22/0x2f [ 9424.161392] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9424.165351] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9424.170207] ? watchdog+0x40/0x40 [ 9424.175192] ? hrtimer_interrupt+0x128/0x2c0 [ 9424.178973] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9424.184199] ? apic_timer_interrupt+0xf/0x20 [ 9424.186228] [ 9424.192795] ? native_safe_halt+0x2e/0x50 [ 9424.194596] kvm_wait+0xcd/0xe0 [ 9424.199566] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9424.203522] do_raw_spin_lock+0x177/0x190 [ 9424.205947] _raw_spin_lock+0x21/0x30 [ 9424.212766] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9424.217433] ? do_raw_spin_unlock+0x75/0x190 [ 9424.221356] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9424.224448] ? do_wait_intr+0xf0/0xf0 [ 9424.226268] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9424.230176] mdc_reint+0x2d/0x110 [mdc] [ 9424.233341] mdc_unlink+0x658/0x870 [mdc] [ 9424.234817] lmv_unlink+0x34d/0xbd0 [lmv] [ 9424.236947] ll_unlink+0x24d/0x7f0 [lustre] [ 9424.238942] vfs_unlink+0x202/0x330 [ 9424.240446] do_unlinkat+0x2fa/0x460 [ 9424.241921] __x64_sys_unlink+0x37/0x50 [ 9424.245139] do_syscall_64+0xc1/0x440 [ 9424.246413] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9424.248519] RIP: 0033:0x7f83e5f5c10b [ 9424.249893] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9424.260929] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9424.267078] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9424.269911] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9424.272492] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9424.274681] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9424.277408] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9425.888059] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6912s! [ 9425.893364] Showing busy workqueues and worker pools: [ 9425.896092] workqueue events: flags=0x0 [ 9425.898888] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9425.903027] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9425.911443] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9425.915385] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9425.921395] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9425.929348] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9425.938690] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9425.943642] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9425.948537] workqueue events_long: flags=0x0 [ 9425.950799] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9425.954657] pending: lu_objects_flush [obdclass] [ 9425.959407] workqueue events_unbound: flags=0x2 [ 9425.962114] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9425.965783] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9425.969472] workqueue events_power_efficient: flags=0x80 [ 9425.972802] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9425.976639] pending: neigh_periodic_work, do_cache_clean [ 9425.979924] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9425.983275] pending: neigh_periodic_work, check_lifetime [ 9425.986562] workqueue mm_percpu_wq: flags=0x8 [ 9425.989028] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9425.992790] pending: vmstat_update, lru_add_drain_per_cpu [ 9425.996414] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9426.000834] pending: vmstat_update [ 9426.003337] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9426.007571] pending: lru_add_drain_per_cpu BAR(48) [ 9426.010357] workqueue ipv6_addrconf: flags=0x40008 [ 9426.013110] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9426.017272] pending: addrconf_verify_work [ 9426.021401] workqueue knbd0-recv: flags=0x1a [ 9426.023913] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9426.027640] in-flight: 650:recv_work [ 9426.030053] workqueue ptlrpc_pinger: flags=0x2 [ 9426.032696] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9426.036329] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9426.039676] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6912s workers=4 idle: 36 466 29757 [ 9426.044064] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9416s workers=2 idle: 652 [ 9428.060554] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9428.067891] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9428.103487] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9428.112684] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9428.119016] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9428.124617] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9428.130199] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9428.140701] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9428.144401] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9428.148215] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9428.154389] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9428.159206] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9428.166788] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9428.173067] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9428.181421] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9428.190039] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9428.195482] Call Trace: [ 9428.197155] [ 9428.198175] ? show_regs.cold.9+0x22/0x2f [ 9428.200464] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9428.205686] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9428.211192] ? watchdog+0x40/0x40 [ 9428.214077] ? hrtimer_interrupt+0x128/0x2c0 [ 9428.217536] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9428.223086] ? apic_timer_interrupt+0xf/0x20 [ 9428.226235] [ 9428.227342] ? native_safe_halt+0x2e/0x50 [ 9428.231333] kvm_wait+0xcd/0xe0 [ 9428.233687] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9428.240714] do_raw_spin_lock+0x177/0x190 [ 9428.242776] _raw_spin_lock+0x21/0x30 [ 9428.245294] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9428.251101] process_one_work+0x2c8/0x700 [ 9428.253232] worker_thread+0x296/0x6e0 [ 9428.259710] ? rescuer_thread+0x570/0x570 [ 9428.261536] kthread+0x1d1/0x200 [ 9428.265126] ? set_kthread_struct+0x70/0x70 [ 9428.269261] ret_from_fork+0x1f/0x30 [ 9448.059968] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 9448.066476] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9448.126473] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9448.145140] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9448.160255] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9448.169885] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9448.190468] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9448.195421] RAX: 0000000000007488 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9448.203458] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9448.210312] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9448.224465] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9448.229258] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9448.232053] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9448.239146] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9448.244227] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9448.250940] Call Trace: [ 9448.252208] [ 9448.253208] ? show_regs.cold.9+0x22/0x2f [ 9448.255985] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9448.258054] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9448.267325] ? watchdog+0x40/0x40 [ 9448.270478] ? hrtimer_interrupt+0x128/0x2c0 [ 9448.272708] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9448.279164] ? apic_timer_interrupt+0xf/0x20 [ 9448.288367] [ 9448.292748] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9448.295207] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9448.299645] do_raw_spin_lock+0x177/0x190 [ 9448.304433] _raw_spin_lock+0x21/0x30 [ 9448.306396] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9448.310351] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9448.313291] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9448.319362] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9448.327139] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9448.334916] ? woken_wake_function+0x30/0x30 [ 9448.337183] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9448.340268] kthread+0x1d1/0x200 [ 9448.342277] ? set_kthread_struct+0x70/0x70 [ 9448.344050] ret_from_fork+0x1f/0x30 [ 9452.056020] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 9452.061786] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9452.062674] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 9452.102431] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9452.102451] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9452.102458] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9452.102477] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9452.102481] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 9452.102487] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9452.102490] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9452.102492] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9452.102494] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9452.102497] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9452.102501] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9452.102504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9452.102507] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9452.102521] Call Trace: [ 9452.102532] [ 9452.102539] ? show_regs.cold.9+0x22/0x2f [ 9452.102550] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9452.102556] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9452.102561] ? watchdog+0x40/0x40 [ 9452.102566] ? hrtimer_interrupt+0x128/0x2c0 [ 9452.102570] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9452.102578] ? apic_timer_interrupt+0xf/0x20 [ 9452.102584] [ 9452.102587] ? native_safe_halt+0x2e/0x50 [ 9452.102591] kvm_wait+0xcd/0xe0 [ 9452.102599] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9452.102606] do_raw_spin_lock+0x177/0x190 [ 9452.102611] _raw_spin_lock+0x21/0x30 [ 9452.102618] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9452.102987] obd_mod_free+0x46/0x140 [ptlrpc] [ 9452.103204] mdc_commit_open+0x91/0xb0 [mdc] [ 9452.103243] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9452.103742] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9452.104130] after_reply+0xe2c/0x1780 [ptlrpc] [ 9452.104355] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9452.104700] ? __next_timer_interrupt+0x160/0x160 [ 9452.104718] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9452.105069] ? do_wait_intr+0xf0/0xf0 [ 9452.105079] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9452.105304] kthread+0x1d1/0x200 [ 9452.105311] ? set_kthread_struct+0x70/0x70 [ 9452.105314] ret_from_fork+0x1f/0x30 [ 9452.301755] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9452.327839] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9452.333237] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9452.339946] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9452.342741] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9452.353615] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9452.358044] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9452.363229] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9452.368331] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9452.372107] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9452.376099] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9452.381166] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9452.385248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9452.388364] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9452.392457] Call Trace: [ 9452.393356] [ 9452.394275] ? show_regs.cold.9+0x22/0x2f [ 9452.396485] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9452.400409] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9452.403209] ? watchdog+0x40/0x40 [ 9452.404755] ? hrtimer_interrupt+0x128/0x2c0 [ 9452.407325] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9452.410638] ? apic_timer_interrupt+0xf/0x20 [ 9452.412871] [ 9452.413856] ? native_safe_halt+0x2e/0x50 [ 9452.417123] kvm_wait+0xcd/0xe0 [ 9452.419389] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9452.423390] do_raw_spin_lock+0x177/0x190 [ 9452.432121] _raw_spin_lock+0x21/0x30 [ 9452.437601] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9452.443762] ? do_raw_spin_unlock+0x75/0x190 [ 9452.449231] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9452.457361] ? do_wait_intr+0xf0/0xf0 [ 9452.460759] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9452.465260] mdc_reint+0x2d/0x110 [mdc] [ 9452.469337] mdc_unlink+0x658/0x870 [mdc] [ 9452.471409] lmv_unlink+0x34d/0xbd0 [lmv] [ 9452.476115] ll_unlink+0x24d/0x7f0 [lustre] [ 9452.480287] vfs_unlink+0x202/0x330 [ 9452.483071] do_unlinkat+0x2fa/0x460 [ 9452.484458] __x64_sys_unlink+0x37/0x50 [ 9452.485914] do_syscall_64+0xc1/0x440 [ 9452.500083] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9452.510055] RIP: 0033:0x7f83e5f5c10b [ 9452.513430] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9452.538125] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9452.545377] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9452.552035] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9452.560700] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9452.569571] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9452.577903] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9456.060013] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9456.062832] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9456.077595] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9456.081092] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9456.084318] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9456.086840] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9456.088987] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9456.094927] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9456.097524] RAX: 0000000000006632 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9456.100006] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9456.102694] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9456.103225] BUG: workqueue lockup - pool [ 9456.105083] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9456.105090] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9456.105094] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9456.105099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9456.105101] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9456.105110] Call Trace: [ 9456.107094] cpus=0-3 [ 9456.109224] [ 9456.109244] ? show_regs.cold.9+0x22/0x2f [ 9456.109257] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9456.109264] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9456.109268] ? watchdog+0x40/0x40 [ 9456.109272] ? hrtimer_interrupt+0x128/0x2c0 [ 9456.109276] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9456.109284] ? apic_timer_interrupt+0xf/0x20 [ 9456.109288] [ 9456.109290] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9456.109297] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9456.109304] do_raw_spin_lock+0x177/0x190 [ 9456.109309] _raw_spin_lock+0x21/0x30 [ 9456.109314] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9456.109694] process_one_work+0x2c8/0x700 [ 9456.109702] worker_thread+0x296/0x6e0 [ 9456.109706] ? rescuer_thread+0x570/0x570 [ 9456.109710] kthread+0x1d1/0x200 [ 9456.109714] ? set_kthread_struct+0x70/0x70 [ 9456.109718] ret_from_fork+0x1f/0x30 [ 9456.165223] flags=0x4 nice=0 stuck for 6942s! [ 9456.166873] Showing busy workqueues and worker pools: [ 9456.168738] workqueue events: flags=0x0 [ 9456.170116] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9456.172706] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9456.177793] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9456.191387] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9456.199077] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9456.214604] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9456.223907] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9456.234403] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9456.241053] workqueue events_long: flags=0x0 [ 9456.244693] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9456.249432] pending: lu_objects_flush [obdclass] [ 9456.256214] workqueue events_unbound: flags=0x2 [ 9456.261235] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9456.265681] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9456.271376] workqueue events_power_efficient: flags=0x80 [ 9456.274723] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9456.278030] pending: neigh_periodic_work, do_cache_clean [ 9456.281727] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9456.288943] pending: neigh_periodic_work, check_lifetime [ 9456.296064] workqueue mm_percpu_wq: flags=0x8 [ 9456.298467] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9456.305105] pending: vmstat_update, lru_add_drain_per_cpu [ 9456.311994] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9456.316278] pending: vmstat_update [ 9456.318340] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9456.324317] pending: lru_add_drain_per_cpu BAR(48) [ 9456.328048] workqueue ipv6_addrconf: flags=0x40008 [ 9456.334769] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9456.341676] pending: addrconf_verify_work [ 9456.344963] workqueue knbd0-recv: flags=0x1a [ 9456.348415] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9456.352393] in-flight: 650:recv_work [ 9456.355051] workqueue ptlrpc_pinger: flags=0x2 [ 9456.357404] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9456.362979] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9456.368328] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6943s workers=4 idle: 36 466 29757 [ 9456.378084] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9446s workers=2 idle: 652 [ 9476.059515] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 9476.063545] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9476.091843] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9476.102646] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9476.109878] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9476.116100] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9476.150692] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9476.169082] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9476.178766] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9476.201502] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9476.213056] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9476.220492] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9476.227825] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9476.232815] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9476.241862] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9476.245853] Call Trace: [ 9476.249234] [ 9476.256164] ? show_regs.cold.9+0x22/0x2f [ 9476.263748] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9476.277659] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9476.290071] ? watchdog+0x40/0x40 [ 9476.291324] ? hrtimer_interrupt+0x128/0x2c0 [ 9476.292790] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9476.303541] ? apic_timer_interrupt+0xf/0x20 [ 9476.306734] [ 9476.309684] ? native_safe_halt+0x2e/0x50 [ 9476.316736] kvm_wait+0xcd/0xe0 [ 9476.320400] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9476.325396] do_raw_spin_lock+0x177/0x190 [ 9476.330871] _raw_spin_lock+0x21/0x30 [ 9476.333721] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9476.338391] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9476.350364] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9476.357914] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9476.360851] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9476.368800] ? woken_wake_function+0x30/0x30 [ 9476.376561] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9476.383880] kthread+0x1d1/0x200 [ 9476.388592] ? set_kthread_struct+0x70/0x70 [ 9476.395608] ret_from_fork+0x1f/0x30 [ 9480.067902] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 9480.083275] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9480.150916] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9480.155731] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9480.161196] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9480.164373] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9480.179584] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9480.184308] RAX: 0000000000007b84 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9480.187478] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9480.194828] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9480.203361] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9480.211679] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9480.215774] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9480.221157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9480.225900] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9480.229669] Call Trace: [ 9480.234876] [ 9480.238561] ? show_regs.cold.9+0x22/0x2f [ 9480.241265] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9480.243713] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9480.250867] ? watchdog+0x40/0x40 [ 9480.254508] ? hrtimer_interrupt+0x128/0x2c0 [ 9480.260733] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9480.263674] ? apic_timer_interrupt+0xf/0x20 [ 9480.266638] [ 9480.267888] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9480.269879] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9480.279226] do_raw_spin_lock+0x177/0x190 [ 9480.280535] _raw_spin_lock+0x21/0x30 [ 9480.282668] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9480.289568] ? do_raw_spin_unlock+0x75/0x190 [ 9480.291551] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9480.295945] ? do_wait_intr+0xf0/0xf0 [ 9480.298810] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9480.307345] mdc_reint+0x2d/0x110 [mdc] [ 9480.311392] mdc_unlink+0x658/0x870 [mdc] [ 9480.317361] lmv_unlink+0x34d/0xbd0 [lmv] [ 9480.320437] ll_unlink+0x24d/0x7f0 [lustre] [ 9480.327936] vfs_unlink+0x202/0x330 [ 9480.333665] do_unlinkat+0x2fa/0x460 [ 9480.338845] __x64_sys_unlink+0x37/0x50 [ 9480.355748] do_syscall_64+0xc1/0x440 [ 9480.357169] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9480.358408] RIP: 0033:0x7f83e5f5c10b [ 9480.359419] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9480.384843] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9480.393944] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9480.401715] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9480.411523] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9480.423926] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9480.431786] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9484.056044] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 9484.060576] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9484.064398] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9484.134144] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9484.134159] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9484.134162] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9484.134175] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9484.134179] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9484.134184] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9484.134186] RDX: 00000000000012fd RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9484.134188] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9484.134190] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9484.134192] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9484.134196] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9484.134198] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9484.134200] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9484.134210] Call Trace: [ 9484.134216] [ 9484.134221] ? show_regs.cold.9+0x22/0x2f [ 9484.134231] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9484.134238] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9484.134242] ? watchdog+0x40/0x40 [ 9484.134246] ? hrtimer_interrupt+0x128/0x2c0 [ 9484.134251] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9484.134260] ? apic_timer_interrupt+0xf/0x20 [ 9484.134265] [ 9484.134268] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9484.134273] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9484.134277] do_raw_spin_lock+0x177/0x190 [ 9484.134283] _raw_spin_lock+0x21/0x30 [ 9484.134290] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9484.144701] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9484.144786] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9484.144792] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9484.144799] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9484.145740] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9484.145753] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9484.145757] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9484.145762] RAX: 0000000000000d31 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9484.145824] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9484.145827] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9484.145829] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9484.145832] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9484.145835] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9484.145838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9484.145841] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9484.145850] Call Trace: [ 9484.145854] [ 9484.145859] ? show_regs.cold.9+0x22/0x2f [ 9484.145868] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9484.145875] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9484.145879] ? watchdog+0x40/0x40 [ 9484.145884] ? hrtimer_interrupt+0x128/0x2c0 [ 9484.145888] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9484.145896] ? apic_timer_interrupt+0xf/0x20 [ 9484.145901] [ 9484.145904] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9484.145909] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9484.145915] do_raw_spin_lock+0x177/0x190 [ 9484.145919] _raw_spin_lock+0x21/0x30 [ 9484.145924] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9484.146643] process_one_work+0x2c8/0x700 [ 9484.146787] worker_thread+0x296/0x6e0 [ 9484.146791] ? rescuer_thread+0x570/0x570 [ 9484.146795] kthread+0x1d1/0x200 [ 9484.146800] ? set_kthread_struct+0x70/0x70 [ 9484.146803] ret_from_fork+0x1f/0x30 [ 9484.515369] obd_mod_free+0x46/0x140 [ptlrpc] [ 9484.517666] mdc_commit_open+0x91/0xb0 [mdc] [ 9484.520262] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9484.522608] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9484.529732] after_reply+0xe2c/0x1780 [ptlrpc] [ 9484.540318] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9484.544450] ? __next_timer_interrupt+0x160/0x160 [ 9484.548183] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9484.551354] ? do_wait_intr+0xf0/0xf0 [ 9484.552860] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9484.560361] kthread+0x1d1/0x200 [ 9484.561340] ? set_kthread_struct+0x70/0x70 [ 9484.563059] ret_from_fork+0x1f/0x30 [ 9486.817732] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6973s! [ 9486.823802] Showing busy workqueues and worker pools: [ 9486.830829] workqueue events: flags=0x0 [ 9486.832514] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9486.835765] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9486.839414] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9486.842564] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9486.846403] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9486.852290] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9486.864115] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9486.870013] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9486.875898] workqueue events_long: flags=0x0 [ 9486.878228] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9486.883406] pending: lu_objects_flush [obdclass] [ 9486.886378] workqueue events_unbound: flags=0x2 [ 9486.888285] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9486.892009] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9486.895629] workqueue events_power_efficient: flags=0x80 [ 9486.899761] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9486.908679] pending: neigh_periodic_work, do_cache_clean [ 9486.923688] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9486.928909] pending: neigh_periodic_work, check_lifetime [ 9486.938349] workqueue mm_percpu_wq: flags=0x8 [ 9486.944678] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9486.963587] pending: vmstat_update, lru_add_drain_per_cpu [ 9486.969847] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9486.983581] pending: vmstat_update [ 9486.984907] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9486.995113] pending: lru_add_drain_per_cpu BAR(48) [ 9486.999978] workqueue ipv6_addrconf: flags=0x40008 [ 9487.003132] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9487.015655] pending: addrconf_verify_work [ 9487.018589] workqueue knbd0-recv: flags=0x1a [ 9487.024537] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9487.029878] in-flight: 650:recv_work [ 9487.032013] workqueue ptlrpc_pinger: flags=0x2 [ 9487.035551] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9487.043807] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9487.052507] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6973s workers=4 idle: 36 466 29757 [ 9487.056784] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9477s workers=2 idle: 652 [ 9504.059027] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 9504.061551] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9504.093499] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9504.104957] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9504.110117] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9504.112102] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9504.125657] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9504.127625] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9504.132503] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9504.143403] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9504.150050] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9504.155141] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9504.159368] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9504.164939] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9504.169184] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9504.171423] Call Trace: [ 9504.172320] [ 9504.173941] ? show_regs.cold.9+0x22/0x2f [ 9504.176708] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9504.179032] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9504.181778] ? watchdog+0x40/0x40 [ 9504.183493] ? hrtimer_interrupt+0x128/0x2c0 [ 9504.185544] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9504.188078] ? apic_timer_interrupt+0xf/0x20 [ 9504.190161] [ 9504.191441] ? native_safe_halt+0x2e/0x50 [ 9504.193379] kvm_wait+0xcd/0xe0 [ 9504.194882] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9504.197328] do_raw_spin_lock+0x177/0x190 [ 9504.201118] _raw_spin_lock+0x21/0x30 [ 9504.204423] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9504.220180] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9504.225471] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9504.228513] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9504.231635] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9504.236233] ? woken_wake_function+0x30/0x30 [ 9504.237920] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9504.243035] kthread+0x1d1/0x200 [ 9504.245216] ? set_kthread_struct+0x70/0x70 [ 9504.247471] ret_from_fork+0x1f/0x30 [ 9508.062045] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 9508.065187] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9508.081300] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9508.085824] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9508.090360] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9508.092981] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9508.100660] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9508.102790] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9508.104972] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9508.107110] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9508.110244] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9508.113172] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9508.116096] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9508.120184] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9508.122688] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9508.125876] Call Trace: [ 9508.126720] [ 9508.127371] ? show_regs.cold.9+0x22/0x2f [ 9508.128665] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9508.130391] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9508.132159] ? watchdog+0x40/0x40 [ 9508.133443] ? hrtimer_interrupt+0x128/0x2c0 [ 9508.135121] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9508.137368] ? apic_timer_interrupt+0xf/0x20 [ 9508.139412] [ 9508.140322] ? native_safe_halt+0x2e/0x50 [ 9508.142227] kvm_wait+0xcd/0xe0 [ 9508.143471] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9508.145018] do_raw_spin_lock+0x177/0x190 [ 9508.146061] _raw_spin_lock+0x21/0x30 [ 9508.147200] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9508.149199] ? do_raw_spin_unlock+0x75/0x190 [ 9508.150372] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9508.152571] ? do_wait_intr+0xf0/0xf0 [ 9508.153694] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9508.155603] mdc_reint+0x2d/0x110 [mdc] [ 9508.158079] mdc_unlink+0x658/0x870 [mdc] [ 9508.159479] lmv_unlink+0x34d/0xbd0 [lmv] [ 9508.160918] ll_unlink+0x24d/0x7f0 [lustre] [ 9508.162347] vfs_unlink+0x202/0x330 [ 9508.163372] do_unlinkat+0x2fa/0x460 [ 9508.164629] __x64_sys_unlink+0x37/0x50 [ 9508.166087] do_syscall_64+0xc1/0x440 [ 9508.167109] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9508.168782] RIP: 0033:0x7f83e5f5c10b [ 9508.170053] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9508.176544] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9508.179091] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9508.181448] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9508.184092] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9508.186472] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9508.189472] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9512.056009] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 9512.058665] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9512.060015] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9512.075442] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9512.075456] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9512.075459] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9512.075473] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9512.075477] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9512.075483] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9512.075485] RDX: 0000000000006ed7 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9512.075488] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9512.075490] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9512.075493] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9512.075496] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9512.075499] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9512.075502] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9512.075510] Call Trace: [ 9512.075515] [ 9512.075521] ? show_regs.cold.9+0x22/0x2f [ 9512.075531] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9512.075538] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9512.075542] ? watchdog+0x40/0x40 [ 9512.075547] ? hrtimer_interrupt+0x128/0x2c0 [ 9512.075551] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9512.075559] ? apic_timer_interrupt+0xf/0x20 [ 9512.075628] [ 9512.075632] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9512.075636] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9512.075641] do_raw_spin_lock+0x177/0x190 [ 9512.075645] _raw_spin_lock+0x21/0x30 [ 9512.075652] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9512.076102] obd_mod_free+0x46/0x140 [ptlrpc] [ 9512.076402] mdc_commit_open+0x91/0xb0 [mdc] [ 9512.076439] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9512.076636] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9512.084863] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9512.085035] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9512.085042] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9512.085049] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9512.085352] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9512.085361] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9512.085365] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9512.085369] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9512.085371] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9512.085373] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9512.085432] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9512.085435] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9512.085438] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9512.085444] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9512.085446] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9512.085550] Call Trace: [ 9512.085558] [ 9512.085563] ? show_regs.cold.9+0x22/0x2f [ 9512.085572] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9512.085577] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9512.085581] ? watchdog+0x40/0x40 [ 9512.085585] ? hrtimer_interrupt+0x128/0x2c0 [ 9512.085589] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9512.085595] ? apic_timer_interrupt+0xf/0x20 [ 9512.085599] [ 9512.085602] ? native_safe_halt+0x2e/0x50 [ 9512.085605] kvm_wait+0xcd/0xe0 [ 9512.085611] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9512.085619] do_raw_spin_lock+0x177/0x190 [ 9512.085623] _raw_spin_lock+0x21/0x30 [ 9512.085627] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9512.085989] process_one_work+0x2c8/0x700 [ 9512.086002] worker_thread+0x296/0x6e0 [ 9512.086007] ? rescuer_thread+0x570/0x570 [ 9512.086010] kthread+0x1d1/0x200 [ 9512.086015] ? set_kthread_struct+0x70/0x70 [ 9512.086018] ret_from_fork+0x1f/0x30 [ 9512.229049] after_reply+0xe2c/0x1780 [ptlrpc] [ 9512.230698] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9512.232524] ? __next_timer_interrupt+0x160/0x160 [ 9512.234074] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9512.235734] ? do_wait_intr+0xf0/0xf0 [ 9512.236660] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9512.238451] kthread+0x1d1/0x200 [ 9512.239501] ? set_kthread_struct+0x70/0x70 [ 9512.241087] ret_from_fork+0x1f/0x30 [ 9517.537116] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7004s! [ 9517.541655] Showing busy workqueues and worker pools: [ 9517.544109] workqueue events: flags=0x0 [ 9517.545742] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9517.548844] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9517.552729] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9517.555944] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9517.559519] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9517.561988] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9517.565252] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9517.567367] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9517.571060] workqueue events_long: flags=0x0 [ 9517.573462] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9517.576317] pending: lu_objects_flush [obdclass] [ 9517.578807] workqueue events_unbound: flags=0x2 [ 9517.580747] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9517.583354] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9517.585576] workqueue events_power_efficient: flags=0x80 [ 9517.587477] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9517.590442] pending: neigh_periodic_work, do_cache_clean [ 9517.594344] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9517.597947] pending: neigh_periodic_work, check_lifetime [ 9517.600406] workqueue mm_percpu_wq: flags=0x8 [ 9517.602667] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9517.605980] pending: vmstat_update, lru_add_drain_per_cpu [ 9517.608747] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9517.612064] pending: vmstat_update [ 9517.613765] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9517.616782] pending: lru_add_drain_per_cpu BAR(48) [ 9517.619125] workqueue ipv6_addrconf: flags=0x40008 [ 9517.621463] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9517.624280] pending: addrconf_verify_work [ 9517.625765] workqueue knbd0-recv: flags=0x1a [ 9517.627742] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9517.630742] in-flight: 650:recv_work [ 9517.632132] workqueue ptlrpc_pinger: flags=0x2 [ 9517.633713] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9517.635665] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9517.638350] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7004s workers=4 idle: 36 466 29757 [ 9517.641125] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9507s workers=2 idle: 652 [ 9532.059158] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9532.063369] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9532.108163] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9532.124501] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9532.131647] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9532.135234] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9532.151129] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9532.153943] RAX: 0000000000000899 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9532.159085] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9532.162839] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9532.167317] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9532.172632] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9532.176575] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9532.183190] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9532.190847] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9532.196377] Call Trace: [ 9532.198172] [ 9532.199451] ? show_regs.cold.9+0x22/0x2f [ 9532.202112] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9532.210598] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9532.215448] ? watchdog+0x40/0x40 [ 9532.218382] ? hrtimer_interrupt+0x128/0x2c0 [ 9532.221070] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9532.225159] ? apic_timer_interrupt+0xf/0x20 [ 9532.227956] [ 9532.229128] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9532.231746] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9532.234504] do_raw_spin_lock+0x177/0x190 [ 9532.236000] _raw_spin_lock+0x21/0x30 [ 9532.238261] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9532.241416] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9532.243969] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9532.246282] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9532.248551] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9532.250528] ? woken_wake_function+0x30/0x30 [ 9532.252199] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9532.254478] kthread+0x1d1/0x200 [ 9532.255467] ? set_kthread_struct+0x70/0x70 [ 9532.257167] ret_from_fork+0x1f/0x30 [ 9536.062254] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 9536.067234] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9536.096186] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9536.109345] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9536.118208] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9536.121677] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9536.136599] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9536.140248] RAX: 0000000000003f88 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9536.144761] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9536.148757] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9536.154707] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9536.160522] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9536.167954] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9536.173074] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9536.179743] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9536.186168] Call Trace: [ 9536.188954] [ 9536.189906] ? show_regs.cold.9+0x22/0x2f [ 9536.192322] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9536.195008] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9536.197934] ? watchdog+0x40/0x40 [ 9536.199878] ? hrtimer_interrupt+0x128/0x2c0 [ 9536.205389] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9536.211385] ? apic_timer_interrupt+0xf/0x20 [ 9536.213351] [ 9536.214825] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9536.217292] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9536.220321] do_raw_spin_lock+0x177/0x190 [ 9536.222601] _raw_spin_lock+0x21/0x30 [ 9536.224513] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9536.229408] ? do_raw_spin_unlock+0x75/0x190 [ 9536.238657] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9536.242873] ? do_wait_intr+0xf0/0xf0 [ 9536.244418] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9536.248719] mdc_reint+0x2d/0x110 [mdc] [ 9536.249896] mdc_unlink+0x658/0x870 [mdc] [ 9536.251233] lmv_unlink+0x34d/0xbd0 [lmv] [ 9536.261609] ll_unlink+0x24d/0x7f0 [lustre] [ 9536.262885] vfs_unlink+0x202/0x330 [ 9536.278177] do_unlinkat+0x2fa/0x460 [ 9536.282484] __x64_sys_unlink+0x37/0x50 [ 9536.286265] do_syscall_64+0xc1/0x440 [ 9536.289180] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9536.291869] RIP: 0033:0x7f83e5f5c10b [ 9536.294115] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9536.318842] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9536.327619] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9536.336245] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9536.342326] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9536.348623] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9536.356964] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9540.060009] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9540.063319] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9540.083882] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9540.088389] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9540.092175] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9540.095500] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 9540.098361] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 9540.105905] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9540.109024] RAX: 00000000000004d1 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9540.112293] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9540.115143] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9540.118393] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9540.120939] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9540.123826] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9540.127385] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9540.129626] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9540.132566] Call Trace: [ 9540.133534] [ 9540.134379] ? show_regs.cold.9+0x22/0x2f [ 9540.135973] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9540.137804] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9540.139710] ? watchdog+0x40/0x40 [ 9540.141164] ? hrtimer_interrupt+0x128/0x2c0 [ 9540.142984] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9540.145052] ? apic_timer_interrupt+0xf/0x20 [ 9540.147232] [ 9540.148094] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 9540.150374] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9540.153057] do_raw_spin_lock+0x177/0x190 [ 9540.155152] _raw_spin_lock+0x21/0x30 [ 9540.156790] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9540.159528] process_one_work+0x2c8/0x700 [ 9540.161389] worker_thread+0x296/0x6e0 [ 9540.162814] ? rescuer_thread+0x570/0x570 [ 9540.164566] kthread+0x1d1/0x200 [ 9540.165736] ? set_kthread_struct+0x70/0x70 [ 9540.167694] ret_from_fork+0x1f/0x30 [ 9544.062984] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 9544.067894] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9544.090751] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9544.095667] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9544.099944] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9544.102238] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9544.109900] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9544.113248] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9544.116111] RDX: 0000000000005d37 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9544.119960] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9544.122384] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9544.125700] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9544.128302] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9544.131510] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9544.134103] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9544.137110] Call Trace: [ 9544.138217] [ 9544.138946] ? show_regs.cold.9+0x22/0x2f [ 9544.140544] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9544.142864] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9544.145494] ? watchdog+0x40/0x40 [ 9544.146928] ? hrtimer_interrupt+0x128/0x2c0 [ 9544.148877] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9544.151027] ? apic_timer_interrupt+0xf/0x20 [ 9544.152927] [ 9544.153829] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9544.156488] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9544.159122] do_raw_spin_lock+0x177/0x190 [ 9544.161405] _raw_spin_lock+0x21/0x30 [ 9544.162779] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9544.165033] obd_mod_free+0x46/0x140 [ptlrpc] [ 9544.167138] mdc_commit_open+0x91/0xb0 [mdc] [ 9544.168994] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9544.171933] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9544.174771] after_reply+0xe2c/0x1780 [ptlrpc] [ 9544.176981] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9544.179783] ? __next_timer_interrupt+0x160/0x160 [ 9544.181766] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9544.184281] ? do_wait_intr+0xf0/0xf0 [ 9544.185765] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9544.187859] kthread+0x1d1/0x200 [ 9544.188788] ? set_kthread_struct+0x70/0x70 [ 9544.190315] ret_from_fork+0x1f/0x30 [ 9547.744166] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7034s! [ 9547.748706] Showing busy workqueues and worker pools: [ 9547.752786] workqueue events: flags=0x0 [ 9547.756277] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9547.763944] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9547.773700] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9547.780177] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9547.785699] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9547.791403] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9547.796848] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9547.803354] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9547.811981] workqueue events_long: flags=0x0 [ 9547.816201] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9547.821098] pending: lu_objects_flush [obdclass] [ 9547.824410] workqueue events_unbound: flags=0x2 [ 9547.828222] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9547.833259] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9547.838858] workqueue events_power_efficient: flags=0x80 [ 9547.845850] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9547.850170] pending: neigh_periodic_work, do_cache_clean [ 9547.855871] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9547.862234] pending: neigh_periodic_work, check_lifetime [ 9547.867144] workqueue mm_percpu_wq: flags=0x8 [ 9547.871955] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9547.878357] pending: vmstat_update, lru_add_drain_per_cpu [ 9547.884132] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9547.890810] pending: vmstat_update [ 9547.893645] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9547.899243] pending: lru_add_drain_per_cpu BAR(48) [ 9547.904383] workqueue ipv6_addrconf: flags=0x40008 [ 9547.910319] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9547.916447] pending: addrconf_verify_work [ 9547.921646] workqueue knbd0-recv: flags=0x1a [ 9547.925928] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9547.932361] in-flight: 650:recv_work [ 9547.936531] workqueue ptlrpc_pinger: flags=0x2 [ 9547.940343] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9547.945357] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9547.951458] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7034s workers=4 idle: 36 466 29757 [ 9547.957765] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9538s workers=2 idle: 652 [ 9560.059032] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9560.065756] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9560.099111] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9560.108739] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9560.113009] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9560.116249] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9560.129715] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9560.135647] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9560.139015] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9560.141555] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9560.147132] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9560.151879] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9560.155665] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9560.171566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9560.177920] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9560.181383] Call Trace: [ 9560.182791] [ 9560.184213] ? show_regs.cold.9+0x22/0x2f [ 9560.185317] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9560.188267] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9560.191110] ? watchdog+0x40/0x40 [ 9560.193722] ? hrtimer_interrupt+0x128/0x2c0 [ 9560.196813] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9560.199036] ? apic_timer_interrupt+0xf/0x20 [ 9560.201276] [ 9560.203089] ? native_safe_halt+0x2e/0x50 [ 9560.206135] kvm_wait+0xcd/0xe0 [ 9560.207017] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9560.208397] do_raw_spin_lock+0x177/0x190 [ 9560.210245] _raw_spin_lock+0x21/0x30 [ 9560.215013] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9560.219941] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9560.228314] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9560.238117] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9560.244892] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9560.249251] ? woken_wake_function+0x30/0x30 [ 9560.255076] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9560.259848] kthread+0x1d1/0x200 [ 9560.261681] ? set_kthread_struct+0x70/0x70 [ 9560.265839] ret_from_fork+0x1f/0x30 [ 9564.062394] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 9564.071697] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9564.120665] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9564.139143] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9564.149443] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9564.156832] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9564.180972] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9564.195604] RAX: 0000000000005e84 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9564.204152] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9564.213364] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9564.215865] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9564.225022] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9564.229858] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9564.243117] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9564.249375] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9564.255964] Call Trace: [ 9564.257080] [ 9564.258035] ? show_regs.cold.9+0x22/0x2f [ 9564.260244] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9564.263781] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9564.267853] ? watchdog+0x40/0x40 [ 9564.269948] ? hrtimer_interrupt+0x128/0x2c0 [ 9564.272240] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9564.274738] ? apic_timer_interrupt+0xf/0x20 [ 9564.277138] [ 9564.278376] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9564.285819] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9564.289142] do_raw_spin_lock+0x177/0x190 [ 9564.291728] _raw_spin_lock+0x21/0x30 [ 9564.293085] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9564.295627] ? do_raw_spin_unlock+0x75/0x190 [ 9564.298829] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9564.301228] ? do_wait_intr+0xf0/0xf0 [ 9564.302752] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9564.304991] mdc_reint+0x2d/0x110 [mdc] [ 9564.306709] mdc_unlink+0x658/0x870 [mdc] [ 9564.308125] lmv_unlink+0x34d/0xbd0 [lmv] [ 9564.309817] ll_unlink+0x24d/0x7f0 [lustre] [ 9564.315451] vfs_unlink+0x202/0x330 [ 9564.320902] do_unlinkat+0x2fa/0x460 [ 9564.322110] __x64_sys_unlink+0x37/0x50 [ 9564.324819] do_syscall_64+0xc1/0x440 [ 9564.326224] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9564.332155] RIP: 0033:0x7f83e5f5c10b [ 9564.333420] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9564.348281] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9564.354307] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9564.361412] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9564.366600] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9564.369918] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9564.375210] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9568.060048] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9568.066625] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9568.104246] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9568.110470] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9568.114889] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9568.118170] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9568.121002] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9568.132452] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9568.140461] RAX: 00000000000076b0 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9568.146242] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9568.152271] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9568.157642] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9568.165418] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9568.172543] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9568.178054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9568.182455] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9568.187401] Call Trace: [ 9568.191649] [ 9568.193476] ? show_regs.cold.9+0x22/0x2f [ 9568.201776] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9568.212437] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9568.219050] ? watchdog+0x40/0x40 [ 9568.221954] ? hrtimer_interrupt+0x128/0x2c0 [ 9568.226418] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9568.229713] ? apic_timer_interrupt+0xf/0x20 [ 9568.231673] [ 9568.233589] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9568.236477] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9568.240356] do_raw_spin_lock+0x177/0x190 [ 9568.242441] _raw_spin_lock+0x21/0x30 [ 9568.246223] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9568.252420] process_one_work+0x2c8/0x700 [ 9568.255459] worker_thread+0x296/0x6e0 [ 9568.259515] ? rescuer_thread+0x570/0x570 [ 9568.262445] kthread+0x1d1/0x200 [ 9568.265189] ? set_kthread_struct+0x70/0x70 [ 9568.268300] ret_from_fork+0x1f/0x30 [ 9572.056054] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 9572.060888] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9572.106420] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9572.114090] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9572.125661] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9572.129065] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9572.155938] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9572.166008] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9572.171040] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9572.176288] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9572.180886] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9572.190717] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9572.194310] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9572.202818] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9572.207983] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9572.213995] Call Trace: [ 9572.214934] [ 9572.220787] ? show_regs.cold.9+0x22/0x2f [ 9572.225087] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9572.231800] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9572.234776] ? watchdog+0x40/0x40 [ 9572.236594] ? hrtimer_interrupt+0x128/0x2c0 [ 9572.238876] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9572.242025] ? apic_timer_interrupt+0xf/0x20 [ 9572.243660] [ 9572.245434] ? native_safe_halt+0x2e/0x50 [ 9572.247622] kvm_wait+0xcd/0xe0 [ 9572.249535] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9572.256473] do_raw_spin_lock+0x177/0x190 [ 9572.259038] _raw_spin_lock+0x21/0x30 [ 9572.262134] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9572.264975] obd_mod_free+0x46/0x140 [ptlrpc] [ 9572.268063] mdc_commit_open+0x91/0xb0 [mdc] [ 9572.271542] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9572.274158] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9572.277627] after_reply+0xe2c/0x1780 [ptlrpc] [ 9572.280633] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9572.283722] ? __next_timer_interrupt+0x160/0x160 [ 9572.288915] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9572.292900] ? do_wait_intr+0xf0/0xf0 [ 9572.294721] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9572.298212] kthread+0x1d1/0x200 [ 9572.300223] ? set_kthread_struct+0x70/0x70 [ 9572.301829] ret_from_fork+0x1f/0x30 [ 9578.465074] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7065s! [ 9578.467833] Showing busy workqueues and worker pools: [ 9578.470526] workqueue events: flags=0x0 [ 9578.472659] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9578.476777] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9578.481898] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9578.486049] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9578.489431] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9578.493724] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9578.498165] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9578.502145] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9578.507808] workqueue events_long: flags=0x0 [ 9578.510131] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9578.513495] pending: lu_objects_flush [obdclass] [ 9578.516187] workqueue events_unbound: flags=0x2 [ 9578.518412] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9578.521612] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9578.524782] workqueue events_power_efficient: flags=0x80 [ 9578.527081] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9578.530398] pending: neigh_periodic_work, do_cache_clean [ 9578.533309] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9578.536651] pending: neigh_periodic_work, check_lifetime [ 9578.539471] workqueue mm_percpu_wq: flags=0x8 [ 9578.541647] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9578.544549] pending: vmstat_update, lru_add_drain_per_cpu [ 9578.547486] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9578.550786] pending: vmstat_update [ 9578.552616] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9578.556139] pending: lru_add_drain_per_cpu BAR(48) [ 9578.558628] workqueue ipv6_addrconf: flags=0x40008 [ 9578.561075] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9578.564778] pending: addrconf_verify_work [ 9578.567562] workqueue knbd0-recv: flags=0x1a [ 9578.569489] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9578.572544] in-flight: 650:recv_work [ 9578.574675] workqueue ptlrpc_pinger: flags=0x2 [ 9578.576603] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9578.579634] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9578.582666] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7065s workers=4 idle: 36 466 29757 [ 9578.586993] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9568s workers=2 idle: 652 [ 9588.059024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9588.063321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9588.086813] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9588.091624] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9588.095564] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9588.098344] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9588.106078] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9588.109333] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9588.112411] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9588.115707] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9588.118833] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9588.123086] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9588.126591] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9588.131726] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9588.134607] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9588.138325] Call Trace: [ 9588.139414] [ 9588.141758] ? show_regs.cold.9+0x22/0x2f [ 9588.143684] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9588.146672] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9588.149521] ? watchdog+0x40/0x40 [ 9588.151451] ? hrtimer_interrupt+0x128/0x2c0 [ 9588.153500] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9588.155986] ? apic_timer_interrupt+0xf/0x20 [ 9588.157962] [ 9588.159928] ? native_safe_halt+0x2e/0x50 [ 9588.161737] kvm_wait+0xcd/0xe0 [ 9588.163443] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9588.165873] do_raw_spin_lock+0x177/0x190 [ 9588.167912] _raw_spin_lock+0x21/0x30 [ 9588.169587] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9588.172726] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9588.175692] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9588.178715] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9588.181446] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9588.184595] ? woken_wake_function+0x30/0x30 [ 9588.187124] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9588.190422] kthread+0x1d1/0x200 [ 9588.192442] ? set_kthread_struct+0x70/0x70 [ 9588.194390] ret_from_fork+0x1f/0x30 [ 9592.062066] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 9592.067343] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9592.112729] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9592.120806] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9592.127091] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 9592.132400] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 9592.143782] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9592.150410] RAX: 000000000000542b RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9592.154984] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9592.160772] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9592.166007] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9592.170620] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9592.175541] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9592.181684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9592.186737] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9592.192089] Call Trace: [ 9592.195374] [ 9592.197117] ? show_regs.cold.9+0x22/0x2f [ 9592.198876] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9592.202389] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9592.204793] ? watchdog+0x40/0x40 [ 9592.206708] ? hrtimer_interrupt+0x128/0x2c0 [ 9592.210481] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9592.213992] ? apic_timer_interrupt+0xf/0x20 [ 9592.217578] [ 9592.223875] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 9592.228095] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9592.231386] do_raw_spin_lock+0x177/0x190 [ 9592.233550] _raw_spin_lock+0x21/0x30 [ 9592.236857] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9592.241805] ? do_raw_spin_unlock+0x75/0x190 [ 9592.244498] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9592.248296] ? do_wait_intr+0xf0/0xf0 [ 9592.250575] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9592.254800] mdc_reint+0x2d/0x110 [mdc] [ 9592.258306] mdc_unlink+0x658/0x870 [mdc] [ 9592.261520] lmv_unlink+0x34d/0xbd0 [lmv] [ 9592.265357] ll_unlink+0x24d/0x7f0 [lustre] [ 9592.270749] vfs_unlink+0x202/0x330 [ 9592.273732] do_unlinkat+0x2fa/0x460 [ 9592.276551] __x64_sys_unlink+0x37/0x50 [ 9592.279894] do_syscall_64+0xc1/0x440 [ 9592.282280] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9592.286488] RIP: 0033:0x7f83e5f5c10b [ 9592.288760] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9592.300037] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9592.305102] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9592.310310] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9592.315275] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9592.321852] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9592.327393] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9596.060749] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 9596.065985] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9596.100391] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9596.106798] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9596.111581] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9596.114404] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9596.118914] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9596.131502] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9596.137105] RAX: 0000000000004da7 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9596.140829] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9596.143324] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9596.147676] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9596.152675] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9596.158321] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9596.163315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9596.165735] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9596.169389] Call Trace: [ 9596.170833] [ 9596.172977] ? show_regs.cold.9+0x22/0x2f [ 9596.175483] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9596.181347] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9596.184228] ? watchdog+0x40/0x40 [ 9596.185147] ? hrtimer_interrupt+0x128/0x2c0 [ 9596.188452] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9596.192864] ? apic_timer_interrupt+0xf/0x20 [ 9596.194404] [ 9596.195918] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9596.197480] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9596.202401] do_raw_spin_lock+0x177/0x190 [ 9596.206032] _raw_spin_lock+0x21/0x30 [ 9596.207737] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9596.214123] process_one_work+0x2c8/0x700 [ 9596.220256] worker_thread+0x296/0x6e0 [ 9596.225766] ? rescuer_thread+0x570/0x570 [ 9596.227131] kthread+0x1d1/0x200 [ 9596.228099] ? set_kthread_struct+0x70/0x70 [ 9596.230488] ret_from_fork+0x1f/0x30 [ 9601.144535] rcu: INFO: rcu_sched self-detected stall on CPU [ 9601.147175] rcu: 0-....: (4793720 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 9601.152228] (t=7620193 jiffies g=262441 q=59524) [ 9601.153910] rcu: rcu_sched kthread starved for 7089381 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 9601.157966] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9601.160602] rcu: RCU grace-period kthread stack dump: [ 9601.162528] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9601.165083] Call Trace: [ 9601.165579] __schedule+0x351/0xcb0 [ 9601.166544] schedule+0xc0/0x180 [ 9601.167306] schedule_timeout+0xb4/0x190 [ 9601.168365] ? do_raw_spin_unlock+0x75/0x190 [ 9601.169717] ? __next_timer_interrupt+0x160/0x160 [ 9601.170911] ? prepare_to_swait_event+0xf3/0x140 [ 9601.172001] rcu_gp_kthread+0x8dd/0x15b0 [ 9601.172923] ? force_qs_rnp+0x2c0/0x2c0 [ 9601.174085] kthread+0x1d1/0x200 [ 9601.175258] ? set_kthread_struct+0x70/0x70 [ 9601.176240] ret_from_fork+0x1f/0x30 [ 9601.177243] rcu: Stack dump where RCU GP kthread last ran: [ 9601.179124] Sending NMI from CPU 0 to CPUs 3: [ 9601.187712] NMI backtrace for cpu 3 [ 9601.187718] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9601.187720] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9601.187721] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9601.187723] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9601.187725] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 [ 9601.187727] RAX: 00000000000067b4 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9601.187728] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9601.187729] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9601.187730] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9601.187731] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9601.187732] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9601.187733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9601.187734] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9601.187735] Call Trace: [ 9601.187736] [ 9601.187736] ? show_regs.cold.9+0x22/0x2f [ 9601.187737] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9601.187738] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9601.187739] ? nmi_handle+0xbd/0x210 [ 9601.187740] ? default_do_nmi+0x4b/0x1b0 [ 9601.187740] ? do_nmi+0x280/0x380 [ 9601.187741] ? end_repeat_nmi+0x16/0x1f [ 9601.187742] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9601.187743] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9601.187744] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9601.187745] [ 9601.187745] do_raw_spin_lock+0x177/0x190 [ 9601.187746] _raw_spin_lock+0x21/0x30 [ 9601.187747] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9601.187748] ? do_raw_spin_unlock+0x75/0x190 [ 9601.187748] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9601.187749] ? do_wait_intr+0xf0/0xf0 [ 9601.187750] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9601.187751] mdc_reint+0x2d/0x110 [mdc] [ 9601.187752] mdc_unlink+0x658/0x870 [mdc] [ 9601.187752] lmv_unlink+0x34d/0xbd0 [lmv] [ 9601.187753] ll_unlink+0x24d/0x7f0 [lustre] [ 9601.187754] vfs_unlink+0x202/0x330 [ 9601.187755] do_unlinkat+0x2fa/0x460 [ 9601.187756] __x64_sys_unlink+0x37/0x50 [ 9601.187757] do_syscall_64+0xc1/0x440 [ 9601.187758] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9601.187759] RIP: 0033:0x7f83e5f5c10b [ 9601.187760] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9601.187762] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9601.187764] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9601.187765] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9601.187766] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9601.187767] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9601.187768] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9601.188448] NMI backtrace for cpu 0 [ 9601.364153] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9601.372001] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9601.375844] Call Trace: [ 9601.377360] [ 9601.378350] ? dump_stack+0xbb/0x10e [ 9601.379761] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9601.381782] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9601.383519] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9601.384993] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9601.387746] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9601.389839] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9601.392275] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9601.393771] ? update_process_times+0x6d/0xc0 [ 9601.395191] ? tick_sched_handle.isra.18+0x27/0x90 [ 9601.397202] ? tick_sched_timer+0x53/0xb0 [ 9601.398929] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9601.400943] ? tick_sched_handle.isra.18+0x90/0x90 [ 9601.403129] ? hrtimer_interrupt+0x128/0x2c0 [ 9601.404886] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9601.406792] ? apic_timer_interrupt+0xf/0x20 [ 9601.408713] [ 9601.409676] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9601.412008] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9601.414643] ? do_raw_spin_lock+0x177/0x190 [ 9601.416307] ? _raw_spin_lock+0x21/0x30 [ 9601.417930] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9601.420549] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9601.422801] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9601.424753] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9601.427143] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9601.429827] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9601.432151] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9601.434853] ? __next_timer_interrupt+0x160/0x160 [ 9601.439627] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9601.441299] ? do_wait_intr+0xf0/0xf0 [ 9601.442635] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9601.444889] ? kthread+0x1d1/0x200 [ 9601.445970] ? set_kthread_struct+0x70/0x70 [ 9601.447427] ? ret_from_fork+0x1f/0x30 [ 9608.679482] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7095s! [ 9608.698254] Showing busy workqueues and worker pools: [ 9608.704139] workqueue events: flags=0x0 [ 9608.711279] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9608.732032] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9608.753242] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9608.767968] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9608.776812] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9608.794479] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9608.821884] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9608.836626] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9608.857442] workqueue events_long: flags=0x0 [ 9608.863303] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9608.868781] pending: lu_objects_flush [obdclass] [ 9608.876755] workqueue events_unbound: flags=0x2 [ 9608.884098] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9608.891813] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9608.900717] workqueue events_power_efficient: flags=0x80 [ 9608.913309] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9608.925480] pending: neigh_periodic_work, do_cache_clean [ 9608.934909] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9608.946772] pending: neigh_periodic_work, check_lifetime [ 9608.955838] workqueue mm_percpu_wq: flags=0x8 [ 9608.957801] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9608.965819] pending: vmstat_update, lru_add_drain_per_cpu [ 9608.969029] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9608.972988] pending: vmstat_update [ 9608.978710] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9608.989904] pending: lru_add_drain_per_cpu BAR(48) [ 9608.994497] workqueue ipv6_addrconf: flags=0x40008 [ 9609.002591] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9609.012477] pending: addrconf_verify_work [ 9609.014734] workqueue knbd0-recv: flags=0x1a [ 9609.024368] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9609.033991] in-flight: 650:recv_work [ 9609.035201] workqueue ptlrpc_pinger: flags=0x2 [ 9609.036494] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9609.047600] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9609.060387] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7095s workers=4 idle: 36 466 29757 [ 9609.077166] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9599s workers=2 idle: 652 [ 9616.064774] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9616.069291] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9616.135290] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9616.151023] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9616.176418] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9616.180360] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9616.197983] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9616.208620] RAX: 0000000000007b0f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9616.222870] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9616.231549] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9616.239496] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9616.251095] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9616.261315] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9616.269619] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9616.275185] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9616.281284] Call Trace: [ 9616.289120] [ 9616.290572] ? show_regs.cold.9+0x22/0x2f [ 9616.300256] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9616.309600] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9616.318116] ? watchdog+0x40/0x40 [ 9616.331529] ? hrtimer_interrupt+0x128/0x2c0 [ 9616.337246] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9616.343561] ? apic_timer_interrupt+0xf/0x20 [ 9616.346482] [ 9616.348103] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9616.351546] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9616.366729] do_raw_spin_lock+0x177/0x190 [ 9616.369534] _raw_spin_lock+0x21/0x30 [ 9616.375001] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9616.381435] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9616.387084] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9616.392304] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9616.396373] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9616.403125] ? woken_wake_function+0x30/0x30 [ 9616.404687] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9616.418872] kthread+0x1d1/0x200 [ 9616.419829] ? set_kthread_struct+0x70/0x70 [ 9616.426527] ret_from_fork+0x1f/0x30 [ 9624.060047] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 9624.067288] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9624.157290] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9624.165362] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9624.175281] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9624.185989] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9624.193272] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9624.215618] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9624.224539] RAX: 0000000000004d8e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9624.236661] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9624.246557] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9624.261150] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9624.266420] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9624.278050] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9624.286783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9624.296676] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9624.302968] Call Trace: [ 9624.304293] [ 9624.305419] ? show_regs.cold.9+0x22/0x2f [ 9624.311223] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9624.316739] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9624.325073] ? watchdog+0x40/0x40 [ 9624.332135] ? hrtimer_interrupt+0x128/0x2c0 [ 9624.338256] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9624.345172] ? apic_timer_interrupt+0xf/0x20 [ 9624.351661] [ 9624.358492] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9624.362236] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9624.365727] do_raw_spin_lock+0x177/0x190 [ 9624.370535] _raw_spin_lock+0x21/0x30 [ 9624.376184] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9624.390478] process_one_work+0x2c8/0x700 [ 9624.402530] worker_thread+0x296/0x6e0 [ 9624.403850] ? rescuer_thread+0x570/0x570 [ 9624.410083] kthread+0x1d1/0x200 [ 9624.414739] ? set_kthread_struct+0x70/0x70 [ 9624.423445] ret_from_fork+0x1f/0x30 [ 9628.062051] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 9628.068336] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9628.107059] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9628.119290] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9628.133071] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9628.136243] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9628.145447] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9628.150216] RAX: 0000000000004e29 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9628.153847] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9628.163162] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9628.166685] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9628.174789] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9628.187620] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9628.195619] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9628.203743] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9628.210013] Call Trace: [ 9628.211649] [ 9628.212371] ? show_regs.cold.9+0x22/0x2f [ 9628.218369] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9628.228339] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9628.230660] ? watchdog+0x40/0x40 [ 9628.232187] ? hrtimer_interrupt+0x128/0x2c0 [ 9628.234186] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9628.239054] ? apic_timer_interrupt+0xf/0x20 [ 9628.249705] [ 9628.251001] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9628.259288] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9628.261172] do_raw_spin_lock+0x177/0x190 [ 9628.265884] _raw_spin_lock+0x21/0x30 [ 9628.268185] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9628.276117] ? do_raw_spin_unlock+0x75/0x190 [ 9628.279278] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9628.287372] ? do_wait_intr+0xf0/0xf0 [ 9628.294134] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9628.298748] mdc_reint+0x2d/0x110 [mdc] [ 9628.301662] mdc_unlink+0x658/0x870 [mdc] [ 9628.307936] lmv_unlink+0x34d/0xbd0 [lmv] [ 9628.310365] ll_unlink+0x24d/0x7f0 [lustre] [ 9628.322492] vfs_unlink+0x202/0x330 [ 9628.324566] do_unlinkat+0x2fa/0x460 [ 9628.325932] __x64_sys_unlink+0x37/0x50 [ 9628.342166] do_syscall_64+0xc1/0x440 [ 9628.343678] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9628.348260] RIP: 0033:0x7f83e5f5c10b [ 9628.351158] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9628.376255] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9628.382872] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9628.388832] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9628.393911] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9628.399965] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9628.405095] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9636.056986] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 9636.061870] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9636.090004] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9636.102259] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9636.106618] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9636.115155] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9636.123125] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9636.126067] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9636.128715] RDX: 0000000000001d28 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9636.132142] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9636.139574] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9636.148075] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9636.151343] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9636.158026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9636.163352] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9636.169581] Call Trace: [ 9636.171455] [ 9636.178344] ? show_regs.cold.9+0x22/0x2f [ 9636.183007] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9636.188473] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9636.192169] ? watchdog+0x40/0x40 [ 9636.198366] ? hrtimer_interrupt+0x128/0x2c0 [ 9636.206187] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9636.217791] ? apic_timer_interrupt+0xf/0x20 [ 9636.224160] [ 9636.224995] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9636.230579] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9636.235239] do_raw_spin_lock+0x177/0x190 [ 9636.238250] _raw_spin_lock+0x21/0x30 [ 9636.242180] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9636.251054] obd_mod_free+0x46/0x140 [ptlrpc] [ 9636.252702] mdc_commit_open+0x91/0xb0 [mdc] [ 9636.260128] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9636.265123] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9636.270122] after_reply+0xe2c/0x1780 [ptlrpc] [ 9636.272294] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9636.275410] ? __next_timer_interrupt+0x160/0x160 [ 9636.277289] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9636.287265] ? do_wait_intr+0xf0/0xf0 [ 9636.288258] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9636.291259] kthread+0x1d1/0x200 [ 9636.291963] ? set_kthread_struct+0x70/0x70 [ 9636.292822] ret_from_fork+0x1f/0x30 [ 9639.393037] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7126s! [ 9639.395802] Showing busy workqueues and worker pools: [ 9639.397317] workqueue events: flags=0x0 [ 9639.398551] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9639.400393] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9639.403324] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9639.405211] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9639.407727] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9639.409597] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9639.412366] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9639.414985] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9639.418539] workqueue events_long: flags=0x0 [ 9639.420143] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9639.422429] pending: lu_objects_flush [obdclass] [ 9639.424403] workqueue events_unbound: flags=0x2 [ 9639.426468] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9639.428717] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9639.430471] workqueue events_power_efficient: flags=0x80 [ 9639.432435] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9639.434809] pending: neigh_periodic_work, do_cache_clean [ 9639.436589] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9639.439438] pending: neigh_periodic_work, check_lifetime [ 9639.442131] workqueue mm_percpu_wq: flags=0x8 [ 9639.443732] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9639.445782] pending: vmstat_update, lru_add_drain_per_cpu [ 9639.447646] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9639.450331] pending: vmstat_update [ 9639.451571] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9639.454162] pending: lru_add_drain_per_cpu BAR(48) [ 9639.455855] workqueue ipv6_addrconf: flags=0x40008 [ 9639.458096] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9639.460294] pending: addrconf_verify_work [ 9639.462065] workqueue knbd0-recv: flags=0x1a [ 9639.463325] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9639.465566] in-flight: 650:recv_work [ 9639.467106] workqueue ptlrpc_pinger: flags=0x2 [ 9639.469037] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9639.471430] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9639.473993] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7126s workers=4 idle: 36 466 29757 [ 9639.477540] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9629s workers=2 idle: 652 [ 9644.059023] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9644.062968] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9644.077050] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9644.080511] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9644.083692] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9644.084879] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9644.091980] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9644.095051] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9644.097476] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9644.100998] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9644.103940] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9644.105707] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9644.109117] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9644.112896] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9644.114421] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9644.117051] Call Trace: [ 9644.118328] [ 9644.118950] ? show_regs.cold.9+0x22/0x2f [ 9644.120851] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9644.122139] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9644.124721] ? watchdog+0x40/0x40 [ 9644.125704] ? hrtimer_interrupt+0x128/0x2c0 [ 9644.126892] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9644.128469] ? apic_timer_interrupt+0xf/0x20 [ 9644.129567] [ 9644.130024] ? native_safe_halt+0x2e/0x50 [ 9644.131243] kvm_wait+0xcd/0xe0 [ 9644.132197] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9644.133705] do_raw_spin_lock+0x177/0x190 [ 9644.134691] _raw_spin_lock+0x21/0x30 [ 9644.136119] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9644.138194] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9644.139980] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9644.141888] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9644.143813] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9644.145492] ? woken_wake_function+0x30/0x30 [ 9644.146984] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9644.148927] kthread+0x1d1/0x200 [ 9644.149877] ? set_kthread_struct+0x70/0x70 [ 9644.151318] ret_from_fork+0x1f/0x30 [ 9652.062843] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 9652.069294] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9652.107231] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9652.117799] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9652.123428] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9652.130497] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9652.133982] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9652.153925] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9652.160221] RAX: 0000000000007f68 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9652.162569] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9652.168171] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9652.172795] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9652.184661] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9652.192016] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9652.201058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9652.204487] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9652.207842] Call Trace: [ 9652.208812] [ 9652.209753] ? show_regs.cold.9+0x22/0x2f [ 9652.211901] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9652.215107] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9652.217539] ? watchdog+0x40/0x40 [ 9652.218815] ? hrtimer_interrupt+0x128/0x2c0 [ 9652.221034] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9652.223646] ? apic_timer_interrupt+0xf/0x20 [ 9652.228089] [ 9652.229625] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9652.235833] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9652.237591] do_raw_spin_lock+0x177/0x190 [ 9652.243345] _raw_spin_lock+0x21/0x30 [ 9652.245119] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9652.252742] process_one_work+0x2c8/0x700 [ 9652.256014] worker_thread+0x296/0x6e0 [ 9652.257775] ? rescuer_thread+0x570/0x570 [ 9652.259012] kthread+0x1d1/0x200 [ 9652.260012] ? set_kthread_struct+0x70/0x70 [ 9652.262340] ret_from_fork+0x1f/0x30 [ 9656.062025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 9656.070935] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9656.107362] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9656.112433] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9656.116915] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9656.120474] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9656.139453] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9656.150573] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9656.159408] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9656.170026] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9656.179559] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9656.183789] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9656.189817] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9656.194439] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9656.200469] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9656.205328] Call Trace: [ 9656.205923] [ 9656.207597] ? show_regs.cold.9+0x22/0x2f [ 9656.210935] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9656.213481] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9656.216470] ? watchdog+0x40/0x40 [ 9656.218347] ? hrtimer_interrupt+0x128/0x2c0 [ 9656.220000] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9656.228790] ? apic_timer_interrupt+0xf/0x20 [ 9656.234197] [ 9656.236782] ? native_safe_halt+0x2e/0x50 [ 9656.240984] kvm_wait+0xcd/0xe0 [ 9656.242978] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9656.247510] do_raw_spin_lock+0x177/0x190 [ 9656.250923] _raw_spin_lock+0x21/0x30 [ 9656.254444] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9656.259206] ? do_raw_spin_unlock+0x75/0x190 [ 9656.261785] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9656.264868] ? do_wait_intr+0xf0/0xf0 [ 9656.266483] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9656.276107] mdc_reint+0x2d/0x110 [mdc] [ 9656.282179] mdc_unlink+0x658/0x870 [mdc] [ 9656.284227] lmv_unlink+0x34d/0xbd0 [lmv] [ 9656.288982] ll_unlink+0x24d/0x7f0 [lustre] [ 9656.296309] vfs_unlink+0x202/0x330 [ 9656.302295] do_unlinkat+0x2fa/0x460 [ 9656.310483] __x64_sys_unlink+0x37/0x50 [ 9656.311927] do_syscall_64+0xc1/0x440 [ 9656.318146] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9656.328318] RIP: 0033:0x7f83e5f5c10b [ 9656.329708] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9656.344897] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9656.348196] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9656.356701] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9656.361092] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9656.364632] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9656.367910] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9664.056129] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 9664.061958] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9664.102371] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9664.108315] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9664.112560] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9664.115620] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9664.125395] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9664.129558] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9664.133576] RDX: 0000000000004a20 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9664.137048] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9664.140859] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9664.145703] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9664.150719] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9664.155681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9664.159707] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9664.164486] Call Trace: [ 9664.166034] [ 9664.167736] ? show_regs.cold.9+0x22/0x2f [ 9664.170516] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9664.172712] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9664.175903] ? watchdog+0x40/0x40 [ 9664.177879] ? hrtimer_interrupt+0x128/0x2c0 [ 9664.180889] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9664.183908] ? apic_timer_interrupt+0xf/0x20 [ 9664.186497] [ 9664.187809] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9664.191572] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9664.194879] do_raw_spin_lock+0x177/0x190 [ 9664.197799] _raw_spin_lock+0x21/0x30 [ 9664.199805] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9664.203447] obd_mod_free+0x46/0x140 [ptlrpc] [ 9664.206804] mdc_commit_open+0x91/0xb0 [mdc] [ 9664.209787] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9664.213649] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9664.217753] after_reply+0xe2c/0x1780 [ptlrpc] [ 9664.221025] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9664.225046] ? __next_timer_interrupt+0x160/0x160 [ 9664.228765] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9664.231539] ? do_wait_intr+0xf0/0xf0 [ 9664.234069] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9664.237868] kthread+0x1d1/0x200 [ 9664.239817] ? set_kthread_struct+0x70/0x70 [ 9664.242859] ret_from_fork+0x1f/0x30 [ 9669.601032] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7156s! [ 9669.605730] Showing busy workqueues and worker pools: [ 9669.607979] workqueue events: flags=0x0 [ 9669.612666] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9669.618168] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9669.623346] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9669.625663] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9669.628251] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9669.631227] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9669.634913] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9669.638321] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9669.642660] workqueue events_long: flags=0x0 [ 9669.644483] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9669.646682] pending: lu_objects_flush [obdclass] [ 9669.648768] workqueue events_unbound: flags=0x2 [ 9669.650530] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9669.653207] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9669.658799] workqueue events_power_efficient: flags=0x80 [ 9669.662276] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9669.666477] pending: neigh_periodic_work, do_cache_clean [ 9669.669349] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9669.672650] pending: neigh_periodic_work, check_lifetime [ 9669.677617] workqueue mm_percpu_wq: flags=0x8 [ 9669.680570] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9669.683894] pending: vmstat_update, lru_add_drain_per_cpu [ 9669.686603] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9669.689498] pending: vmstat_update [ 9669.692096] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9669.695435] pending: lru_add_drain_per_cpu BAR(48) [ 9669.698169] workqueue ipv6_addrconf: flags=0x40008 [ 9669.700335] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9669.703266] pending: addrconf_verify_work [ 9669.706771] workqueue knbd0-recv: flags=0x1a [ 9669.711255] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9669.714023] in-flight: 650:recv_work [ 9669.715828] workqueue ptlrpc_pinger: flags=0x2 [ 9669.720303] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9669.725064] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9669.728421] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7156s workers=4 idle: 36 466 29757 [ 9669.732178] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9659s workers=2 idle: 652 [ 9672.059476] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9672.070625] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9672.124307] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9672.135964] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9672.145373] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9672.149684] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9672.165019] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 9672.170793] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9672.174944] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9672.181371] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9672.188897] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9672.195557] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9672.201032] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9672.207444] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9672.211516] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9672.217035] Call Trace: [ 9672.218796] [ 9672.220299] ? show_regs.cold.9+0x22/0x2f [ 9672.223664] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9672.227552] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9672.230500] ? watchdog+0x40/0x40 [ 9672.231705] ? hrtimer_interrupt+0x128/0x2c0 [ 9672.236118] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9672.242532] ? apic_timer_interrupt+0xf/0x20 [ 9672.246927] [ 9672.248784] ? native_safe_halt+0x2e/0x50 [ 9672.252602] kvm_wait+0xcd/0xe0 [ 9672.255500] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9672.259937] do_raw_spin_lock+0x177/0x190 [ 9672.261498] _raw_spin_lock+0x21/0x30 [ 9672.264656] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9672.272943] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9672.278871] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9672.286535] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9672.293656] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9672.297801] ? woken_wake_function+0x30/0x30 [ 9672.302498] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9672.309690] kthread+0x1d1/0x200 [ 9672.313344] ? set_kthread_struct+0x70/0x70 [ 9672.320341] ret_from_fork+0x1f/0x30 [ 9680.060094] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 9680.063588] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9680.083734] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9680.088363] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9680.093039] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9680.096324] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9680.097805] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9680.103231] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9680.105367] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9680.107242] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9680.109730] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9680.114729] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9680.117892] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9680.121415] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9680.125247] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9680.127922] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9680.131034] Call Trace: [ 9680.131886] [ 9680.133733] ? show_regs.cold.9+0x22/0x2f [ 9680.136726] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9680.139934] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9680.142459] ? watchdog+0x40/0x40 [ 9680.143362] ? hrtimer_interrupt+0x128/0x2c0 [ 9680.144803] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9680.146991] ? apic_timer_interrupt+0xf/0x20 [ 9680.149337] [ 9680.150534] ? native_safe_halt+0x2e/0x50 [ 9680.152807] kvm_wait+0xcd/0xe0 [ 9680.157583] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9680.162659] do_raw_spin_lock+0x177/0x190 [ 9680.166535] _raw_spin_lock+0x21/0x30 [ 9680.168631] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9680.170956] process_one_work+0x2c8/0x700 [ 9680.172455] worker_thread+0x296/0x6e0 [ 9680.179548] ? rescuer_thread+0x570/0x570 [ 9680.181034] kthread+0x1d1/0x200 [ 9680.184186] ? set_kthread_struct+0x70/0x70 [ 9680.186559] ret_from_fork+0x1f/0x30 [ 9684.063063] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 9684.071819] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9684.116118] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9684.127636] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9684.138301] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9684.140701] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9684.150377] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9684.153639] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9684.156591] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9684.164491] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9684.169763] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9684.174766] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9684.178694] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9684.181294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9684.184095] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9684.189677] Call Trace: [ 9684.191718] [ 9684.192464] ? show_regs.cold.9+0x22/0x2f [ 9684.194263] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9684.201544] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9684.204581] ? watchdog+0x40/0x40 [ 9684.206521] ? hrtimer_interrupt+0x128/0x2c0 [ 9684.216386] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9684.220988] ? apic_timer_interrupt+0xf/0x20 [ 9684.222987] [ 9684.223802] ? native_safe_halt+0x2e/0x50 [ 9684.226521] kvm_wait+0xcd/0xe0 [ 9684.228377] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9684.232522] do_raw_spin_lock+0x177/0x190 [ 9684.234573] _raw_spin_lock+0x21/0x30 [ 9684.237206] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9684.241047] ? do_raw_spin_unlock+0x75/0x190 [ 9684.242497] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9684.246322] ? do_wait_intr+0xf0/0xf0 [ 9684.247719] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9684.250769] mdc_reint+0x2d/0x110 [mdc] [ 9684.252516] mdc_unlink+0x658/0x870 [mdc] [ 9684.254797] lmv_unlink+0x34d/0xbd0 [lmv] [ 9684.256067] ll_unlink+0x24d/0x7f0 [lustre] [ 9684.258045] vfs_unlink+0x202/0x330 [ 9684.260130] do_unlinkat+0x2fa/0x460 [ 9684.261675] __x64_sys_unlink+0x37/0x50 [ 9684.263921] do_syscall_64+0xc1/0x440 [ 9684.265570] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9684.268140] RIP: 0033:0x7f83e5f5c10b [ 9684.269406] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9684.278029] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9684.282189] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9684.285657] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9684.290509] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9684.294326] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9684.297599] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9696.058296] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 9696.069145] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9696.102249] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9696.108874] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9696.116727] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9696.122308] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9696.140192] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9696.145502] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9696.150460] RDX: 0000000000007f86 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9696.162007] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9696.173219] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9696.177382] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9696.180658] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9696.190937] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9696.195378] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9696.198714] Call Trace: [ 9696.199645] [ 9696.200513] ? show_regs.cold.9+0x22/0x2f [ 9696.203214] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9696.207382] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9696.213340] ? watchdog+0x40/0x40 [ 9696.218726] ? hrtimer_interrupt+0x128/0x2c0 [ 9696.220350] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9696.226259] ? apic_timer_interrupt+0xf/0x20 [ 9696.228748] [ 9696.230212] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9696.232323] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9696.235558] do_raw_spin_lock+0x177/0x190 [ 9696.239588] _raw_spin_lock+0x21/0x30 [ 9696.247458] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9696.255325] obd_mod_free+0x46/0x140 [ptlrpc] [ 9696.261736] mdc_commit_open+0x91/0xb0 [mdc] [ 9696.264404] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9696.267523] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9696.276269] after_reply+0xe2c/0x1780 [ptlrpc] [ 9696.278990] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9696.280818] ? __next_timer_interrupt+0x160/0x160 [ 9696.282734] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9696.287984] ? do_wait_intr+0xf0/0xf0 [ 9696.289513] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9696.296376] kthread+0x1d1/0x200 [ 9696.297792] ? set_kthread_struct+0x70/0x70 [ 9696.300038] ret_from_fork+0x1f/0x30 [ 9699.808089] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7186s! [ 9699.812480] Showing busy workqueues and worker pools: [ 9699.814524] workqueue events: flags=0x0 [ 9699.816902] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9699.820692] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9699.826816] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9699.830620] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9699.835622] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9699.840535] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9699.847402] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9699.851972] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9699.857660] workqueue events_long: flags=0x0 [ 9699.860264] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9699.864198] pending: lu_objects_flush [obdclass] [ 9699.867123] workqueue events_unbound: flags=0x2 [ 9699.869520] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9699.872740] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9699.875845] workqueue events_power_efficient: flags=0x80 [ 9699.878083] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9699.881456] pending: neigh_periodic_work, do_cache_clean [ 9699.884220] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9699.887530] pending: neigh_periodic_work, check_lifetime [ 9699.890355] workqueue mm_percpu_wq: flags=0x8 [ 9699.892583] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9699.895705] pending: vmstat_update, lru_add_drain_per_cpu [ 9699.898467] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9699.901573] pending: vmstat_update [ 9699.905118] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9699.909926] pending: lru_add_drain_per_cpu BAR(48) [ 9699.912902] workqueue ipv6_addrconf: flags=0x40008 [ 9699.914900] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9699.917942] pending: addrconf_verify_work [ 9699.919989] workqueue knbd0-recv: flags=0x1a [ 9699.921993] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9699.924993] in-flight: 650:recv_work [ 9699.926793] workqueue ptlrpc_pinger: flags=0x2 [ 9699.928789] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9699.931715] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9699.934738] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7186s workers=4 idle: 36 466 29757 [ 9699.938780] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9689s workers=2 idle: 652 [ 9700.059035] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9700.064416] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9700.106377] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9700.123183] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9700.135614] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9700.139722] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9700.156415] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9700.169174] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9700.176874] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9700.183811] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9700.197525] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9700.208843] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9700.220642] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9700.234545] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9700.245568] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9700.257330] Call Trace: [ 9700.262133] [ 9700.262914] ? show_regs.cold.9+0x22/0x2f [ 9700.266809] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9700.279104] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9700.282245] ? watchdog+0x40/0x40 [ 9700.284543] ? hrtimer_interrupt+0x128/0x2c0 [ 9700.288851] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9700.292994] ? apic_timer_interrupt+0xf/0x20 [ 9700.300558] [ 9700.301839] ? native_safe_halt+0x2e/0x50 [ 9700.304300] kvm_wait+0xcd/0xe0 [ 9700.307356] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9700.314570] do_raw_spin_lock+0x177/0x190 [ 9700.316920] _raw_spin_lock+0x21/0x30 [ 9700.321935] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9700.329363] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9700.333363] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9700.337189] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9700.340169] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9700.346853] ? woken_wake_function+0x30/0x30 [ 9700.351912] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9700.356463] kthread+0x1d1/0x200 [ 9700.359513] ? set_kthread_struct+0x70/0x70 [ 9700.367471] ret_from_fork+0x1f/0x30 [ 9708.060571] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 9708.067820] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9708.114750] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9708.122377] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9708.126659] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9708.129408] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9708.131960] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9708.144341] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9708.151496] RAX: 0000000000007988 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9708.164109] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9708.177142] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9708.179714] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9708.187957] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9708.197685] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9708.203347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9708.211084] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9708.226766] Call Trace: [ 9708.231412] [ 9708.232163] ? show_regs.cold.9+0x22/0x2f [ 9708.238154] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9708.242882] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9708.245771] ? watchdog+0x40/0x40 [ 9708.257188] ? hrtimer_interrupt+0x128/0x2c0 [ 9708.258509] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9708.260169] ? apic_timer_interrupt+0xf/0x20 [ 9708.270382] [ 9708.271353] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9708.274637] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9708.287654] do_raw_spin_lock+0x177/0x190 [ 9708.292900] _raw_spin_lock+0x21/0x30 [ 9708.294577] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9708.297854] process_one_work+0x2c8/0x700 [ 9708.300736] worker_thread+0x296/0x6e0 [ 9708.302746] ? rescuer_thread+0x570/0x570 [ 9708.307665] kthread+0x1d1/0x200 [ 9708.309919] ? set_kthread_struct+0x70/0x70 [ 9708.312024] ret_from_fork+0x1f/0x30 [ 9712.062427] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 9712.076251] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9712.118246] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9712.122572] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9712.126229] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9712.129237] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9712.137129] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9712.140176] RAX: 00000000000062d5 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9712.150421] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9712.172782] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9712.175055] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9712.189229] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9712.194300] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9712.199173] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9712.201359] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9712.204384] Call Trace: [ 9712.206189] [ 9712.207409] ? show_regs.cold.9+0x22/0x2f [ 9712.211245] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9712.214131] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9712.216464] ? watchdog+0x40/0x40 [ 9712.218307] ? hrtimer_interrupt+0x128/0x2c0 [ 9712.221036] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9712.223266] ? apic_timer_interrupt+0xf/0x20 [ 9712.225404] [ 9712.226520] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9712.229111] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9712.231938] do_raw_spin_lock+0x177/0x190 [ 9712.234466] _raw_spin_lock+0x21/0x30 [ 9712.236849] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9712.240532] ? do_raw_spin_unlock+0x75/0x190 [ 9712.244132] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9712.246361] ? do_wait_intr+0xf0/0xf0 [ 9712.247593] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9712.257967] mdc_reint+0x2d/0x110 [mdc] [ 9712.260854] mdc_unlink+0x658/0x870 [mdc] [ 9712.263687] lmv_unlink+0x34d/0xbd0 [lmv] [ 9712.266519] ll_unlink+0x24d/0x7f0 [lustre] [ 9712.271922] vfs_unlink+0x202/0x330 [ 9712.274786] do_unlinkat+0x2fa/0x460 [ 9712.279221] __x64_sys_unlink+0x37/0x50 [ 9712.283081] do_syscall_64+0xc1/0x440 [ 9712.285547] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9712.291928] RIP: 0033:0x7f83e5f5c10b [ 9712.294933] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9712.309784] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9712.317202] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9712.322097] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9712.325560] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9712.330282] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9712.334249] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9724.057801] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 9724.064850] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9724.101562] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9724.111472] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9724.119255] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9724.125952] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9724.141394] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9724.148841] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9724.153421] RDX: 0000000000003de9 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9724.157653] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9724.162993] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9724.168914] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9724.174377] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9724.177258] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9724.180276] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9724.183600] Call Trace: [ 9724.193742] [ 9724.195003] ? show_regs.cold.9+0x22/0x2f [ 9724.204159] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9724.209810] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9724.215219] ? watchdog+0x40/0x40 [ 9724.217314] ? hrtimer_interrupt+0x128/0x2c0 [ 9724.222416] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9724.226633] ? apic_timer_interrupt+0xf/0x20 [ 9724.230380] [ 9724.234468] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9724.237793] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9724.242282] do_raw_spin_lock+0x177/0x190 [ 9724.245655] _raw_spin_lock+0x21/0x30 [ 9724.247915] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9724.251036] obd_mod_free+0x46/0x140 [ptlrpc] [ 9724.257769] mdc_commit_open+0x91/0xb0 [mdc] [ 9724.265811] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9724.275168] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9724.282758] after_reply+0xe2c/0x1780 [ptlrpc] [ 9724.286536] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9724.293085] ? __next_timer_interrupt+0x160/0x160 [ 9724.300196] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9724.303408] ? do_wait_intr+0xf0/0xf0 [ 9724.305176] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9724.309183] kthread+0x1d1/0x200 [ 9724.312420] ? set_kthread_struct+0x70/0x70 [ 9724.319449] ret_from_fork+0x1f/0x30 [ 9728.064935] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9728.075734] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9728.176716] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9728.198463] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9728.207526] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9728.212492] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9728.248411] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9728.257498] RAX: 000000000000483d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9728.267938] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9728.283732] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9728.298437] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9728.309799] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9728.316383] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9728.330502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9728.340211] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9728.354210] Call Trace: [ 9728.355751] [ 9728.357646] ? show_regs.cold.9+0x22/0x2f [ 9728.360374] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9728.366479] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9728.372213] ? watchdog+0x40/0x40 [ 9728.380289] ? hrtimer_interrupt+0x128/0x2c0 [ 9728.388131] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9728.390510] ? apic_timer_interrupt+0xf/0x20 [ 9728.405581] [ 9728.411304] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9728.419419] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9728.428639] do_raw_spin_lock+0x177/0x190 [ 9728.434479] _raw_spin_lock+0x21/0x30 [ 9728.438508] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9728.447750] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9728.452168] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9728.455659] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9728.464649] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9728.477258] ? woken_wake_function+0x30/0x30 [ 9728.480537] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9728.496648] kthread+0x1d1/0x200 [ 9728.504755] ? set_kthread_struct+0x70/0x70 [ 9728.509959] ret_from_fork+0x1f/0x30 [ 9730.017110] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7216s! [ 9730.021909] Showing busy workqueues and worker pools: [ 9730.023761] workqueue events: flags=0x0 [ 9730.026274] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9730.029670] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9730.034516] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9730.037686] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9730.041118] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9730.044837] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9730.049131] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9730.052603] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9730.060455] workqueue events_long: flags=0x0 [ 9730.064627] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9730.067724] pending: lu_objects_flush [obdclass] [ 9730.071548] workqueue events_unbound: flags=0x2 [ 9730.074684] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9730.081625] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9730.085340] workqueue events_power_efficient: flags=0x80 [ 9730.088037] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9730.090943] pending: neigh_periodic_work, do_cache_clean [ 9730.096563] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9730.101162] pending: neigh_periodic_work, check_lifetime [ 9730.106099] workqueue mm_percpu_wq: flags=0x8 [ 9730.109729] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9730.113407] pending: vmstat_update, lru_add_drain_per_cpu [ 9730.118424] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9730.122890] pending: vmstat_update [ 9730.126901] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9730.129275] pending: lru_add_drain_per_cpu BAR(48) [ 9730.133478] workqueue ipv6_addrconf: flags=0x40008 [ 9730.141024] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9730.145007] pending: addrconf_verify_work [ 9730.147091] workqueue knbd0-recv: flags=0x1a [ 9730.153959] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9730.159764] in-flight: 650:recv_work [ 9730.164018] workqueue ptlrpc_pinger: flags=0x2 [ 9730.169970] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9730.172086] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9730.177663] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7216s workers=4 idle: 36 466 29757 [ 9730.183947] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9720s workers=2 idle: 652 [ 9736.063075] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9736.072414] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9736.090695] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9736.094142] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9736.097348] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9736.100085] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9736.102045] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9736.108204] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9736.110753] RAX: 0000000000002021 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9736.112933] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9736.115376] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9736.117321] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9736.119336] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9736.121336] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9736.123614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9736.125417] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9736.127539] Call Trace: [ 9736.128336] [ 9736.129028] ? show_regs.cold.9+0x22/0x2f [ 9736.130349] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9736.132136] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9736.134431] ? watchdog+0x40/0x40 [ 9736.135581] ? hrtimer_interrupt+0x128/0x2c0 [ 9736.137069] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9736.138410] ? apic_timer_interrupt+0xf/0x20 [ 9736.139978] [ 9736.140654] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9736.142518] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9736.144271] do_raw_spin_lock+0x177/0x190 [ 9736.145557] _raw_spin_lock+0x21/0x30 [ 9736.146687] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9736.148590] process_one_work+0x2c8/0x700 [ 9736.149875] worker_thread+0x296/0x6e0 [ 9736.151216] ? rescuer_thread+0x570/0x570 [ 9736.152564] kthread+0x1d1/0x200 [ 9736.153765] ? set_kthread_struct+0x70/0x70 [ 9736.155191] ret_from_fork+0x1f/0x30 [ 9740.062628] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 9740.066783] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9740.090652] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9740.094994] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9740.099609] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9740.102424] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9740.110388] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9740.114447] RAX: 00000000000061c5 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9740.117332] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9740.120346] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9740.123287] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9740.126023] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9740.128837] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9740.132572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9740.134837] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9740.137729] Call Trace: [ 9740.138766] [ 9740.139519] ? show_regs.cold.9+0x22/0x2f [ 9740.140968] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9740.142918] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9740.144540] ? watchdog+0x40/0x40 [ 9740.145877] ? hrtimer_interrupt+0x128/0x2c0 [ 9740.147534] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9740.149062] ? apic_timer_interrupt+0xf/0x20 [ 9740.150980] [ 9740.152049] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9740.153866] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9740.155937] do_raw_spin_lock+0x177/0x190 [ 9740.157899] _raw_spin_lock+0x21/0x30 [ 9740.159395] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9740.161661] ? do_raw_spin_unlock+0x75/0x190 [ 9740.163540] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9740.165871] ? do_wait_intr+0xf0/0xf0 [ 9740.167460] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9740.169532] mdc_reint+0x2d/0x110 [mdc] [ 9740.170969] mdc_unlink+0x658/0x870 [mdc] [ 9740.173066] lmv_unlink+0x34d/0xbd0 [lmv] [ 9740.175177] ll_unlink+0x24d/0x7f0 [lustre] [ 9740.176888] vfs_unlink+0x202/0x330 [ 9740.178318] do_unlinkat+0x2fa/0x460 [ 9740.179915] __x64_sys_unlink+0x37/0x50 [ 9740.181239] do_syscall_64+0xc1/0x440 [ 9740.182592] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9740.184496] RIP: 0033:0x7f83e5f5c10b [ 9740.185995] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9740.192575] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9740.195911] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9740.198382] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9740.200974] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9740.203454] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9740.205929] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9756.056420] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [ 9756.059404] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 9756.060134] Modules linked in: lustre(O) [ 9756.067367] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9756.067457] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9756.067468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9756.067471] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9756.067659] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9756.067665] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9756.067670] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9756.067673] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9756.067675] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9756.067678] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9756.067680] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9756.067683] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9756.067687] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9756.067689] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9756.067697] Call Trace: [ 9756.067710] [ 9756.067716] ? show_regs.cold.9+0x22/0x2f [ 9756.067726] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9756.067732] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9756.067737] ? watchdog+0x40/0x40 [ 9756.067742] ? hrtimer_interrupt+0x128/0x2c0 [ 9756.067746] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9756.067753] ? apic_timer_interrupt+0xf/0x20 [ 9756.067758] [ 9756.067761] ? native_safe_halt+0x2e/0x50 [ 9756.067765] kvm_wait+0xcd/0xe0 [ 9756.067772] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9756.067780] do_raw_spin_lock+0x177/0x190 [ 9756.067785] _raw_spin_lock+0x21/0x30 [ 9756.067789] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9756.068211] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9756.068433] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9756.070246] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9756.070327] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9756.070334] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9756.070336] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9756.070346] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9756.070350] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9756.070355] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9756.070357] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9756.070359] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9756.070362] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9756.070364] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9756.070368] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9756.070375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9756.070378] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9756.070385] Call Trace: [ 9756.070389] [ 9756.070393] ? show_regs.cold.9+0x22/0x2f [ 9756.070403] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9756.070409] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9756.070413] ? watchdog+0x40/0x40 [ 9756.070417] ? hrtimer_interrupt+0x128/0x2c0 [ 9756.070421] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9756.070433] ? apic_timer_interrupt+0xf/0x20 [ 9756.070438] [ 9756.070441] ? native_safe_halt+0x2e/0x50 [ 9756.070444] kvm_wait+0xcd/0xe0 [ 9756.070453] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9756.070460] do_raw_spin_lock+0x177/0x190 [ 9756.070464] _raw_spin_lock+0x21/0x30 [ 9756.070516] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9756.070856] obd_mod_free+0x46/0x140 [ptlrpc] [ 9756.071173] mdc_commit_open+0x91/0xb0 [mdc] [ 9756.071213] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9756.071699] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9756.071906] after_reply+0xe2c/0x1780 [ptlrpc] [ 9756.072375] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9756.101703] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9756.102102] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9756.112778] ? __next_timer_interrupt+0x160/0x160 [ 9756.112803] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9756.120868] ? woken_wake_function+0x30/0x30 [ 9756.120891] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9756.125431] ? do_wait_intr+0xf0/0xf0 [ 9756.125453] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9756.125796] kthread+0x1d1/0x200 [ 9756.125802] ? set_kthread_struct+0x70/0x70 [ 9756.125806] ret_from_fork+0x1f/0x30 [ 9756.328350] kthread+0x1d1/0x200 [ 9756.329482] ? set_kthread_struct+0x70/0x70 [ 9756.330989] ret_from_fork+0x1f/0x30 [ 9760.226046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7247s! [ 9760.230366] Showing busy workqueues and worker pools: [ 9760.232906] workqueue events: flags=0x0 [ 9760.234696] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9760.236981] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9760.257470] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9760.262188] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9760.271564] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9760.279418] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9760.286642] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9760.291245] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9760.299779] workqueue events_long: flags=0x0 [ 9760.307037] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9760.309147] pending: lu_objects_flush [obdclass] [ 9760.317702] workqueue events_unbound: flags=0x2 [ 9760.319251] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9760.333781] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9760.339856] workqueue events_power_efficient: flags=0x80 [ 9760.347650] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9760.356765] pending: neigh_periodic_work, do_cache_clean [ 9760.362753] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9760.366605] pending: neigh_periodic_work, check_lifetime [ 9760.372858] workqueue mm_percpu_wq: flags=0x8 [ 9760.376090] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9760.381679] pending: vmstat_update, lru_add_drain_per_cpu [ 9760.392873] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9760.398574] pending: vmstat_update [ 9760.405434] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9760.409869] pending: lru_add_drain_per_cpu BAR(48) [ 9760.413065] workqueue ipv6_addrconf: flags=0x40008 [ 9760.415569] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9760.418840] pending: addrconf_verify_work [ 9760.426643] workqueue knbd0-recv: flags=0x1a [ 9760.431053] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9760.433084] in-flight: 650:recv_work [ 9760.434533] workqueue ptlrpc_pinger: flags=0x2 [ 9760.445944] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9760.455476] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9760.459650] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7247s workers=4 idle: 36 466 29757 [ 9760.465829] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9750s workers=2 idle: 652 [ 9764.061544] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9764.069818] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9764.116560] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9764.131806] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9764.146925] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9764.152431] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9764.161319] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9764.187186] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9764.191130] RAX: 0000000000007e6e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9764.200608] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9764.208978] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9764.213932] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9764.224101] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9764.238794] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9764.248439] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9764.259154] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9764.270216] Call Trace: [ 9764.271170] [ 9764.273040] ? show_regs.cold.9+0x22/0x2f [ 9764.280294] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9764.285014] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9764.298136] ? watchdog+0x40/0x40 [ 9764.302507] ? hrtimer_interrupt+0x128/0x2c0 [ 9764.311654] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9764.315617] ? apic_timer_interrupt+0xf/0x20 [ 9764.319522] [ 9764.322402] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9764.324361] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9764.333426] do_raw_spin_lock+0x177/0x190 [ 9764.338966] _raw_spin_lock+0x21/0x30 [ 9764.344821] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9764.350958] process_one_work+0x2c8/0x700 [ 9764.358354] worker_thread+0x296/0x6e0 [ 9764.360508] ? rescuer_thread+0x570/0x570 [ 9764.361961] kthread+0x1d1/0x200 [ 9764.363170] ? set_kthread_struct+0x70/0x70 [ 9764.384785] ret_from_fork+0x1f/0x30 [ 9768.062015] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 9768.065193] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9768.087738] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9768.090976] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9768.093696] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9768.095164] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9768.104517] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9768.107933] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9768.114809] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9768.117011] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9768.124291] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9768.126612] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9768.128314] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9768.133739] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9768.136787] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9768.138707] Call Trace: [ 9768.140024] [ 9768.142361] ? show_regs.cold.9+0x22/0x2f [ 9768.143504] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9768.145046] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9768.146102] ? watchdog+0x40/0x40 [ 9768.147918] ? hrtimer_interrupt+0x128/0x2c0 [ 9768.149682] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9768.151641] ? apic_timer_interrupt+0xf/0x20 [ 9768.156079] [ 9768.157005] ? native_safe_halt+0x2e/0x50 [ 9768.158878] kvm_wait+0xcd/0xe0 [ 9768.165633] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9768.168966] do_raw_spin_lock+0x177/0x190 [ 9768.170574] _raw_spin_lock+0x21/0x30 [ 9768.171975] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9768.173814] ? do_raw_spin_unlock+0x75/0x190 [ 9768.175434] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9768.177994] ? do_wait_intr+0xf0/0xf0 [ 9768.179455] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9768.181250] mdc_reint+0x2d/0x110 [mdc] [ 9768.183287] mdc_unlink+0x658/0x870 [mdc] [ 9768.184792] lmv_unlink+0x34d/0xbd0 [lmv] [ 9768.188988] ll_unlink+0x24d/0x7f0 [lustre] [ 9768.192157] vfs_unlink+0x202/0x330 [ 9768.194746] do_unlinkat+0x2fa/0x460 [ 9768.197317] __x64_sys_unlink+0x37/0x50 [ 9768.199107] do_syscall_64+0xc1/0x440 [ 9768.201068] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9768.204286] RIP: 0033:0x7f83e5f5c10b [ 9768.207742] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9768.216121] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9768.219840] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9768.223563] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9768.227091] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9768.233547] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9768.238437] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9781.148540] rcu: INFO: rcu_sched self-detected stall on CPU [ 9781.153485] rcu: 0-....: (4913511 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [ 9781.160022] (t=7800199 jiffies g=262441 q=59552) [ 9781.162132] rcu: rcu_sched kthread starved for 7269390 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 9781.171714] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9781.180732] rcu: RCU grace-period kthread stack dump: [ 9781.187633] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9781.199590] Call Trace: [ 9781.200540] __schedule+0x351/0xcb0 [ 9781.206571] schedule+0xc0/0x180 [ 9781.207839] schedule_timeout+0xb4/0x190 [ 9781.211340] ? do_raw_spin_unlock+0x75/0x190 [ 9781.217200] ? __next_timer_interrupt+0x160/0x160 [ 9781.223484] ? prepare_to_swait_event+0xf3/0x140 [ 9781.226472] rcu_gp_kthread+0x8dd/0x15b0 [ 9781.238400] ? force_qs_rnp+0x2c0/0x2c0 [ 9781.240755] kthread+0x1d1/0x200 [ 9781.241903] ? set_kthread_struct+0x70/0x70 [ 9781.247714] ret_from_fork+0x1f/0x30 [ 9781.248945] rcu: Stack dump where RCU GP kthread last ran: [ 9781.254951] NMI backtrace for cpu 0 [ 9781.264438] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9781.271577] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9781.277012] Call Trace: [ 9781.278336] [ 9781.279140] ? dump_stack+0xbb/0x10e [ 9781.281187] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9781.283861] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9781.286162] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9781.288425] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9781.290776] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9781.293093] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 9781.295745] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 9781.298086] ? update_process_times+0x6d/0xc0 [ 9781.300023] ? tick_sched_handle.isra.18+0x27/0x90 [ 9781.302120] ? tick_sched_timer+0x53/0xb0 [ 9781.303997] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9781.306047] ? tick_sched_handle.isra.18+0x90/0x90 [ 9781.308231] ? hrtimer_interrupt+0x128/0x2c0 [ 9781.309955] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9781.312169] ? apic_timer_interrupt+0xf/0x20 [ 9781.314340] [ 9781.315221] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9781.317330] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9781.319923] ? do_raw_spin_lock+0x177/0x190 [ 9781.321773] ? _raw_spin_lock+0x21/0x30 [ 9781.323275] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9781.325297] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9781.327276] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9781.329066] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9781.330997] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9781.332850] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9781.334450] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9781.335990] ? __next_timer_interrupt+0x160/0x160 [ 9781.337701] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9781.339496] ? do_wait_intr+0xf0/0xf0 [ 9781.340626] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9781.342507] ? kthread+0x1d1/0x200 [ 9781.343559] ? set_kthread_struct+0x70/0x70 [ 9781.344652] ? ret_from_fork+0x1f/0x30 [ 9781.345723] NMI backtrace for cpu 0 [ 9781.346775] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9781.350155] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9781.352693] Call Trace: [ 9781.353450] [ 9781.354100] ? dump_stack+0xbb/0x10e [ 9781.355438] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9781.356936] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9781.358606] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9781.360002] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9781.361604] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9781.363392] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9781.365023] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9781.366827] ? update_process_times+0x6d/0xc0 [ 9781.368374] ? tick_sched_handle.isra.18+0x27/0x90 [ 9781.369934] ? tick_sched_timer+0x53/0xb0 [ 9781.371448] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9781.373010] ? tick_sched_handle.isra.18+0x90/0x90 [ 9781.374875] ? hrtimer_interrupt+0x128/0x2c0 [ 9781.376226] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9781.377792] ? apic_timer_interrupt+0xf/0x20 [ 9781.379170] [ 9781.379784] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9781.381595] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9781.383161] ? do_raw_spin_lock+0x177/0x190 [ 9781.384585] ? _raw_spin_lock+0x21/0x30 [ 9781.385846] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9781.387715] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9781.389501] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9781.391013] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9781.393154] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9781.395147] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9781.396977] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9781.399113] ? __next_timer_interrupt+0x160/0x160 [ 9781.400615] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9781.402482] ? do_wait_intr+0xf0/0xf0 [ 9781.403858] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9781.405838] ? kthread+0x1d1/0x200 [ 9781.407130] ? set_kthread_struct+0x70/0x70 [ 9781.408678] ? ret_from_fork+0x1f/0x30 [ 9784.061575] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 9784.070232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9784.112266] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9784.124785] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9784.130147] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9784.133170] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9784.143852] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9784.148370] RAX: 0000000000007b65 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9784.150338] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9784.163088] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9784.171693] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9784.177023] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9784.178844] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9784.188389] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9784.190991] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9784.196538] Call Trace: [ 9784.197851] [ 9784.198607] ? show_regs.cold.9+0x22/0x2f [ 9784.202008] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9784.204830] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9784.210195] ? watchdog+0x40/0x40 [ 9784.211717] ? hrtimer_interrupt+0x128/0x2c0 [ 9784.216759] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9784.220629] ? apic_timer_interrupt+0xf/0x20 [ 9784.224831] [ 9784.226802] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9784.230900] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9784.233037] do_raw_spin_lock+0x177/0x190 [ 9784.234796] _raw_spin_lock+0x21/0x30 [ 9784.236104] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9784.238915] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9784.241535] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9784.243758] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9784.246516] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9784.248514] ? woken_wake_function+0x30/0x30 [ 9784.249848] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9784.252384] kthread+0x1d1/0x200 [ 9784.253511] ? set_kthread_struct+0x70/0x70 [ 9784.255216] ret_from_fork+0x1f/0x30 [ 9790.953644] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7277s! [ 9790.963307] Showing busy workqueues and worker pools: [ 9790.974293] workqueue events: flags=0x0 [ 9790.975269] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9790.976787] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9790.991512] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9791.008620] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9791.014479] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9791.020834] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9791.028847] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9791.032327] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9791.044887] workqueue events_long: flags=0x0 [ 9791.048395] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9791.057446] pending: lu_objects_flush [obdclass] [ 9791.062056] workqueue events_unbound: flags=0x2 [ 9791.067720] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9791.073859] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9791.078504] workqueue events_power_efficient: flags=0x80 [ 9791.082593] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9791.089476] pending: neigh_periodic_work, do_cache_clean [ 9791.095824] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9791.103463] pending: neigh_periodic_work, check_lifetime [ 9791.108098] workqueue mm_percpu_wq: flags=0x8 [ 9791.110672] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9791.117248] pending: vmstat_update, lru_add_drain_per_cpu [ 9791.120376] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9791.127056] pending: vmstat_update [ 9791.128570] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9791.134493] pending: lru_add_drain_per_cpu BAR(48) [ 9791.140920] workqueue ipv6_addrconf: flags=0x40008 [ 9791.146329] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9791.153063] pending: addrconf_verify_work [ 9791.157482] workqueue knbd0-recv: flags=0x1a [ 9791.165648] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9791.169794] in-flight: 650:recv_work [ 9791.174250] workqueue ptlrpc_pinger: flags=0x2 [ 9791.178932] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9791.182991] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9791.188508] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7278s workers=4 idle: 36 466 29757 [ 9791.197346] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9781s workers=2 idle: 652 [ 9792.063342] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9792.072567] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9792.119075] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9792.125066] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9792.129660] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9792.133078] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9792.135358] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9792.144752] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9792.148773] RAX: 0000000000001a7f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9792.154330] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9792.158540] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9792.161106] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9792.165737] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9792.169712] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9792.173753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9792.176860] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9792.180756] Call Trace: [ 9792.182245] [ 9792.186445] ? show_regs.cold.9+0x22/0x2f [ 9792.188109] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9792.193904] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9792.201574] ? watchdog+0x40/0x40 [ 9792.205994] ? hrtimer_interrupt+0x128/0x2c0 [ 9792.209630] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9792.216514] ? apic_timer_interrupt+0xf/0x20 [ 9792.221668] [ 9792.223050] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9792.228718] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9792.237153] do_raw_spin_lock+0x177/0x190 [ 9792.240733] _raw_spin_lock+0x21/0x30 [ 9792.242457] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9792.247616] process_one_work+0x2c8/0x700 [ 9792.252082] worker_thread+0x296/0x6e0 [ 9792.254216] ? rescuer_thread+0x570/0x570 [ 9792.258658] kthread+0x1d1/0x200 [ 9792.266393] ? set_kthread_struct+0x70/0x70 [ 9792.271017] ret_from_fork+0x1f/0x30 [ 9796.062022] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 9796.064539] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9796.091782] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9796.100007] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9796.110475] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9796.114669] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9796.127705] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9796.133457] RAX: 0000000000004920 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9796.137116] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9796.141048] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9796.146548] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9796.149919] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9796.154678] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9796.159347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9796.162589] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9796.169361] Call Trace: [ 9796.174661] [ 9796.175522] ? show_regs.cold.9+0x22/0x2f [ 9796.179381] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9796.182120] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9796.184306] ? watchdog+0x40/0x40 [ 9796.185947] ? hrtimer_interrupt+0x128/0x2c0 [ 9796.188039] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9796.190063] ? apic_timer_interrupt+0xf/0x20 [ 9796.191815] [ 9796.192709] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9796.194980] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9796.199030] do_raw_spin_lock+0x177/0x190 [ 9796.202514] _raw_spin_lock+0x21/0x30 [ 9796.208170] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9796.210766] ? do_raw_spin_unlock+0x75/0x190 [ 9796.220321] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9796.225711] ? do_wait_intr+0xf0/0xf0 [ 9796.228107] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9796.232726] mdc_reint+0x2d/0x110 [mdc] [ 9796.234825] mdc_unlink+0x658/0x870 [mdc] [ 9796.237199] lmv_unlink+0x34d/0xbd0 [lmv] [ 9796.239205] ll_unlink+0x24d/0x7f0 [lustre] [ 9796.241722] vfs_unlink+0x202/0x330 [ 9796.243366] do_unlinkat+0x2fa/0x460 [ 9796.245539] __x64_sys_unlink+0x37/0x50 [ 9796.247260] do_syscall_64+0xc1/0x440 [ 9796.249047] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9796.251650] RIP: 0033:0x7f83e5f5c10b [ 9796.253501] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9796.262946] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9796.267052] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9796.270696] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9796.280743] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9796.286075] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9796.293417] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9812.059015] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 9812.062868] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9812.083127] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9812.087268] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9812.092112] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9812.095078] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9812.103421] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9812.106394] RAX: 000000000000718e RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9812.109581] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9812.112578] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9812.115929] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9812.118574] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9812.120921] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9812.124196] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9812.126755] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9812.129911] Call Trace: [ 9812.130957] [ 9812.132017] ? show_regs.cold.9+0x22/0x2f [ 9812.133480] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9812.135656] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9812.137403] ? watchdog+0x40/0x40 [ 9812.138597] ? hrtimer_interrupt+0x128/0x2c0 [ 9812.140414] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9812.142696] ? apic_timer_interrupt+0xf/0x20 [ 9812.144565] [ 9812.145540] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9812.147496] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9812.149931] do_raw_spin_lock+0x177/0x190 [ 9812.151554] _raw_spin_lock+0x21/0x30 [ 9812.153079] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9812.155998] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9812.158526] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9812.160803] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9812.163525] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9812.165639] ? woken_wake_function+0x30/0x30 [ 9812.167503] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9812.170115] kthread+0x1d1/0x200 [ 9812.171600] ? set_kthread_struct+0x70/0x70 [ 9812.173696] ret_from_fork+0x1f/0x30 [ 9816.058159] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 9816.070043] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9816.126252] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9816.136996] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9816.142598] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9816.149074] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9816.163046] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9816.170388] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9816.176657] RDX: 00000000000001ba RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9816.185891] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9816.192404] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9816.199040] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9816.202593] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9816.210373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9816.215652] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9816.219290] Call Trace: [ 9816.221321] [ 9816.222450] ? show_regs.cold.9+0x22/0x2f [ 9816.224629] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9816.226836] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9816.228858] ? watchdog+0x40/0x40 [ 9816.234491] ? hrtimer_interrupt+0x128/0x2c0 [ 9816.239326] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9816.242995] ? apic_timer_interrupt+0xf/0x20 [ 9816.245015] [ 9816.245910] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9816.250690] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9816.255718] do_raw_spin_lock+0x177/0x190 [ 9816.257770] _raw_spin_lock+0x21/0x30 [ 9816.260366] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9816.266659] obd_mod_free+0x46/0x140 [ptlrpc] [ 9816.278096] mdc_commit_open+0x91/0xb0 [mdc] [ 9816.279575] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9816.285554] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9816.290653] after_reply+0xe2c/0x1780 [ptlrpc] [ 9816.297437] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9816.301866] ? __next_timer_interrupt+0x160/0x160 [ 9816.312579] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9816.320066] ? do_wait_intr+0xf0/0xf0 [ 9816.323752] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9816.330481] kthread+0x1d1/0x200 [ 9816.333240] ? set_kthread_struct+0x70/0x70 [ 9816.337359] ret_from_fork+0x1f/0x30 [ 9820.062111] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9820.072709] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9820.117423] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9820.123206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9820.130338] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9820.136404] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9820.141867] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9820.155971] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9820.165923] RAX: 0000000000007a8f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9820.169124] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9820.175372] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9820.180345] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9820.187859] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9820.192557] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9820.198169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9820.203287] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9820.205155] Call Trace: [ 9820.205660] [ 9820.206209] ? show_regs.cold.9+0x22/0x2f [ 9820.209241] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9820.210518] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9820.214633] ? watchdog+0x40/0x40 [ 9820.218387] ? hrtimer_interrupt+0x128/0x2c0 [ 9820.221673] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9820.224397] ? apic_timer_interrupt+0xf/0x20 [ 9820.231049] [ 9820.232501] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9820.240076] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9820.246528] do_raw_spin_lock+0x177/0x190 [ 9820.252604] _raw_spin_lock+0x21/0x30 [ 9820.255825] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9820.259868] process_one_work+0x2c8/0x700 [ 9820.262664] worker_thread+0x296/0x6e0 [ 9820.265314] ? rescuer_thread+0x570/0x570 [ 9820.270420] kthread+0x1d1/0x200 [ 9820.272528] ? set_kthread_struct+0x70/0x70 [ 9820.274798] ret_from_fork+0x1f/0x30 [ 9821.669222] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7308s! [ 9821.677632] Showing busy workqueues and worker pools: [ 9821.682639] workqueue events: flags=0x0 [ 9821.684323] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9821.687614] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9821.692773] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9821.696097] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9821.698501] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9821.701891] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9821.705745] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9821.709447] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9821.713670] workqueue events_long: flags=0x0 [ 9821.715858] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9821.719634] pending: lu_objects_flush [obdclass] [ 9821.722162] workqueue events_unbound: flags=0x2 [ 9821.724443] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9821.727889] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9821.730659] workqueue events_power_efficient: flags=0x80 [ 9821.733774] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9821.737651] pending: neigh_periodic_work, do_cache_clean [ 9821.740899] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9821.744447] pending: neigh_periodic_work, check_lifetime [ 9821.747577] workqueue mm_percpu_wq: flags=0x8 [ 9821.750017] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9821.753954] pending: vmstat_update, lru_add_drain_per_cpu [ 9821.756470] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9821.760346] pending: vmstat_update [ 9821.762455] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9821.766436] pending: lru_add_drain_per_cpu BAR(48) [ 9821.769451] workqueue ipv6_addrconf: flags=0x40008 [ 9821.772121] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9821.775549] pending: addrconf_verify_work [ 9821.777692] workqueue knbd0-recv: flags=0x1a [ 9821.779609] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9821.783396] in-flight: 650:recv_work [ 9821.785659] workqueue ptlrpc_pinger: flags=0x2 [ 9821.787814] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9821.791731] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9821.795737] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7308s workers=4 idle: 36 466 29757 [ 9821.800646] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9811s workers=2 idle: 652 [ 9824.062029] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 9824.065184] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9824.096205] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9824.103002] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9824.108385] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9824.111587] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9824.124348] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9824.130092] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9824.133612] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9824.137648] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9824.142556] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9824.157107] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9824.165423] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9824.175477] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9824.186567] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9824.198359] Call Trace: [ 9824.200539] [ 9824.203437] ? show_regs.cold.9+0x22/0x2f [ 9824.211298] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9824.214432] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9824.216510] ? watchdog+0x40/0x40 [ 9824.218976] ? hrtimer_interrupt+0x128/0x2c0 [ 9824.226520] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9824.230292] ? apic_timer_interrupt+0xf/0x20 [ 9824.235533] [ 9824.237071] ? native_safe_halt+0x2e/0x50 [ 9824.238131] kvm_wait+0xcd/0xe0 [ 9824.239029] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9824.243359] do_raw_spin_lock+0x177/0x190 [ 9824.248468] _raw_spin_lock+0x21/0x30 [ 9824.250225] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9824.258612] ? do_raw_spin_unlock+0x75/0x190 [ 9824.264218] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9824.266059] ? do_wait_intr+0xf0/0xf0 [ 9824.268617] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9824.271639] mdc_reint+0x2d/0x110 [mdc] [ 9824.273168] mdc_unlink+0x658/0x870 [mdc] [ 9824.274681] lmv_unlink+0x34d/0xbd0 [lmv] [ 9824.276285] ll_unlink+0x24d/0x7f0 [lustre] [ 9824.278593] vfs_unlink+0x202/0x330 [ 9824.279923] do_unlinkat+0x2fa/0x460 [ 9824.281110] __x64_sys_unlink+0x37/0x50 [ 9824.282269] do_syscall_64+0xc1/0x440 [ 9824.283415] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9824.285195] RIP: 0033:0x7f83e5f5c10b [ 9824.286572] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9824.292205] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9824.294929] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9824.297363] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9824.300320] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9824.303022] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9824.305837] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9840.062755] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 9840.075159] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9840.119552] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9840.125685] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9840.129915] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9840.132071] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9840.140208] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9840.143906] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9840.147077] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9840.150321] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9840.154074] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9840.157508] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9840.160908] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9840.175398] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9840.177787] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9840.181152] Call Trace: [ 9840.182207] [ 9840.183118] ? show_regs.cold.9+0x22/0x2f [ 9840.185005] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9840.187136] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9840.189156] ? watchdog+0x40/0x40 [ 9840.190657] ? hrtimer_interrupt+0x128/0x2c0 [ 9840.192525] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9840.194685] ? apic_timer_interrupt+0xf/0x20 [ 9840.196553] [ 9840.197324] ? native_safe_halt+0x2e/0x50 [ 9840.198966] kvm_wait+0xcd/0xe0 [ 9840.200204] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9840.202214] do_raw_spin_lock+0x177/0x190 [ 9840.203513] _raw_spin_lock+0x21/0x30 [ 9840.205426] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9840.208497] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9840.211104] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9840.213720] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9840.216221] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9840.218924] ? woken_wake_function+0x30/0x30 [ 9840.220877] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9840.223734] kthread+0x1d1/0x200 [ 9840.225464] ? set_kthread_struct+0x70/0x70 [ 9840.227377] ret_from_fork+0x1f/0x30 [ 9848.056121] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 9848.063321] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9848.068492] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9848.068586] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9848.068594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9848.068597] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9848.068612] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9848.068616] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9848.068620] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9848.068622] RDX: 0000000000003ef0 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9848.068625] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9848.068627] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9848.068630] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9848.068633] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9848.068636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9848.068638] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9848.068646] Call Trace: [ 9848.068652] [ 9848.068657] ? show_regs.cold.9+0x22/0x2f [ 9848.068666] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9848.068673] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9848.068677] ? watchdog+0x40/0x40 [ 9848.068682] ? hrtimer_interrupt+0x128/0x2c0 [ 9848.068692] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9848.068700] ? apic_timer_interrupt+0xf/0x20 [ 9848.068705] [ 9848.068709] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9848.068714] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9848.068719] do_raw_spin_lock+0x177/0x190 [ 9848.068724] _raw_spin_lock+0x21/0x30 [ 9848.068731] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9848.073258] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [ 9848.137827] obd_mod_free+0x46/0x140 [ptlrpc] [ 9848.155550] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9848.155616] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9848.155624] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9848.155630] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9848.155919] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9848.155931] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9848.155935] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9848.155939] RAX: 00000000000076de RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9848.155942] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9848.155944] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9848.155946] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9848.155948] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9848.155951] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9848.155953] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9848.155955] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9848.155963] Call Trace: [ 9848.155969] [ 9848.155974] ? show_regs.cold.9+0x22/0x2f [ 9848.155984] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9848.155991] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9848.155996] ? watchdog+0x40/0x40 [ 9848.156001] ? hrtimer_interrupt+0x128/0x2c0 [ 9848.156005] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9848.156012] ? apic_timer_interrupt+0xf/0x20 [ 9848.156016] [ 9848.156020] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9848.156024] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9848.156029] do_raw_spin_lock+0x177/0x190 [ 9848.156033] _raw_spin_lock+0x21/0x30 [ 9848.156038] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9848.156264] process_one_work+0x2c8/0x700 [ 9848.156271] worker_thread+0x296/0x6e0 [ 9848.156276] ? rescuer_thread+0x570/0x570 [ 9848.156280] kthread+0x1d1/0x200 [ 9848.156284] ? set_kthread_struct+0x70/0x70 [ 9848.156287] ret_from_fork+0x1f/0x30 [ 9848.653480] mdc_commit_open+0x91/0xb0 [mdc] [ 9848.662310] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9848.673527] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9848.681089] after_reply+0xe2c/0x1780 [ptlrpc] [ 9848.690934] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9848.700054] ? __next_timer_interrupt+0x160/0x160 [ 9848.712146] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9848.716975] ? do_wait_intr+0xf0/0xf0 [ 9848.718197] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9848.740623] kthread+0x1d1/0x200 [ 9848.747405] ? set_kthread_struct+0x70/0x70 [ 9848.750171] ret_from_fork+0x1f/0x30 [ 9851.881411] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7338s! [ 9851.888884] Showing busy workqueues and worker pools: [ 9851.897108] workqueue events: flags=0x0 [ 9851.909356] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9851.912477] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9851.921577] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9851.932993] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9851.936946] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9851.942948] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9851.945919] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9851.951843] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9851.971436] workqueue events_long: flags=0x0 [ 9851.983354] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9851.991868] pending: lu_objects_flush [obdclass] [ 9851.999019] workqueue events_unbound: flags=0x2 [ 9852.000775] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9852.006936] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9852.017633] workqueue events_power_efficient: flags=0x80 [ 9852.024412] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9852.031117] pending: neigh_periodic_work, do_cache_clean [ 9852.039305] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9852.048488] pending: neigh_periodic_work, check_lifetime [ 9852.055019] workqueue mm_percpu_wq: flags=0x8 [ 9852.058519] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9852.062550] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 9852.065651] pending: vmstat_update, lru_add_drain_per_cpu [ 9852.065803] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9852.065818] pending: vmstat_update [ 9852.065824] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9852.065831] pending: lru_add_drain_per_cpu BAR(48) [ 9852.065856] workqueue ipv6_addrconf: flags=0x40008 [ 9852.065859] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9852.065866] pending: addrconf_verify_work [ 9852.065878] workqueue knbd0-recv: flags=0x1a [ 9852.065881] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9852.065889] in-flight: 650:recv_work [ 9852.065898] workqueue ptlrpc_pinger: flags=0x2 [ 9852.065901] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9852.065906] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9852.068434] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7338s workers=4 idle: 36 466 29757 [ 9852.068454] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9842s workers=2 idle: 652 [ 9852.182921] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9852.245348] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9852.263538] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9852.272637] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9852.276617] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9852.300775] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9852.307877] RAX: 000000000000360d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9852.313895] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9852.321798] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9852.332249] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9852.340884] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9852.351505] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9852.361876] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9852.369686] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9852.378599] Call Trace: [ 9852.384574] [ 9852.386985] ? show_regs.cold.9+0x22/0x2f [ 9852.401792] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9852.415315] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9852.423165] ? watchdog+0x40/0x40 [ 9852.430516] ? hrtimer_interrupt+0x128/0x2c0 [ 9852.432193] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9852.441846] ? apic_timer_interrupt+0xf/0x20 [ 9852.449139] [ 9852.450514] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9852.460836] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9852.476432] do_raw_spin_lock+0x177/0x190 [ 9852.485799] _raw_spin_lock+0x21/0x30 [ 9852.492558] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9852.498103] ? do_raw_spin_unlock+0x75/0x190 [ 9852.504713] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9852.516166] ? do_wait_intr+0xf0/0xf0 [ 9852.527349] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9852.535869] mdc_reint+0x2d/0x110 [mdc] [ 9852.540738] mdc_unlink+0x658/0x870 [mdc] [ 9852.546970] lmv_unlink+0x34d/0xbd0 [lmv] [ 9852.558055] ll_unlink+0x24d/0x7f0 [lustre] [ 9852.565145] vfs_unlink+0x202/0x330 [ 9852.569874] do_unlinkat+0x2fa/0x460 [ 9852.577650] __x64_sys_unlink+0x37/0x50 [ 9852.580117] do_syscall_64+0xc1/0x440 [ 9852.585956] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9852.588884] RIP: 0033:0x7f83e5f5c10b [ 9852.594105] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9852.620518] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9852.628023] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9852.638201] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9852.642911] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9852.647700] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9852.656882] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9868.059642] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 9868.062599] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9868.094765] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9868.106160] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9868.111024] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9868.116148] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9868.137529] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9868.142639] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9868.146620] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9868.151600] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9868.155761] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9868.163625] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9868.172636] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9868.176244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9868.180360] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9868.186397] Call Trace: [ 9868.187596] [ 9868.189581] ? show_regs.cold.9+0x22/0x2f [ 9868.193839] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9868.198516] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9868.201031] ? watchdog+0x40/0x40 [ 9868.203299] ? hrtimer_interrupt+0x128/0x2c0 [ 9868.204826] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9868.210244] ? apic_timer_interrupt+0xf/0x20 [ 9868.211845] [ 9868.216646] ? native_safe_halt+0x2e/0x50 [ 9868.218181] kvm_wait+0xcd/0xe0 [ 9868.221143] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9868.226630] do_raw_spin_lock+0x177/0x190 [ 9868.229196] _raw_spin_lock+0x21/0x30 [ 9868.230583] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9868.240407] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9868.248349] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9868.253078] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9868.257749] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9868.260000] ? woken_wake_function+0x30/0x30 [ 9868.265088] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9868.268895] kthread+0x1d1/0x200 [ 9868.271438] ? set_kthread_struct+0x70/0x70 [ 9868.274727] ret_from_fork+0x1f/0x30 [ 9876.062064] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9876.071908] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9876.101849] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9876.107664] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9876.111632] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9876.114706] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9876.117850] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9876.126128] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9876.129480] RAX: 0000000000000b85 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9876.132635] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9876.136057] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9876.139571] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9876.142717] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9876.145919] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9876.150453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9876.153205] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9876.159599] Call Trace: [ 9876.161785] [ 9876.162576] ? show_regs.cold.9+0x22/0x2f [ 9876.164624] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9876.166544] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9876.168848] ? watchdog+0x40/0x40 [ 9876.170423] ? hrtimer_interrupt+0x128/0x2c0 [ 9876.173218] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9876.176873] ? apic_timer_interrupt+0xf/0x20 [ 9876.178881] [ 9876.179857] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9876.182532] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9876.184743] do_raw_spin_lock+0x177/0x190 [ 9876.185958] _raw_spin_lock+0x21/0x30 [ 9876.187826] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9876.190792] process_one_work+0x2c8/0x700 [ 9876.193089] worker_thread+0x296/0x6e0 [ 9876.194875] ? rescuer_thread+0x570/0x570 [ 9876.196871] kthread+0x1d1/0x200 [ 9876.198599] ? set_kthread_struct+0x70/0x70 [ 9876.203850] ret_from_fork+0x1f/0x30 [ 9880.059489] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 9880.062752] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 9880.063581] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9880.063657] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9880.063664] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9880.063669] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9880.063683] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9880.063686] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9880.063691] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9880.063693] RDX: 00000000000035e6 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9880.063695] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9880.063697] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9880.063699] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9880.063702] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9880.063705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9880.063707] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9880.063714] Call Trace: [ 9880.063720] [ 9880.063725] ? show_regs.cold.9+0x22/0x2f [ 9880.063734] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9880.063746] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9880.063750] ? watchdog+0x40/0x40 [ 9880.063754] ? hrtimer_interrupt+0x128/0x2c0 [ 9880.063758] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9880.063767] ? apic_timer_interrupt+0xf/0x20 [ 9880.063771] [ 9880.063774] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9880.063779] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9880.063784] do_raw_spin_lock+0x177/0x190 [ 9880.063788] _raw_spin_lock+0x21/0x30 [ 9880.063794] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9880.064943] obd_mod_free+0x46/0x140 [ptlrpc] [ 9880.065195] mdc_commit_open+0x91/0xb0 [mdc] [ 9880.065230] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9880.065773] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9880.066075] after_reply+0xe2c/0x1780 [ptlrpc] [ 9880.066290] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9880.066645] ? __next_timer_interrupt+0x160/0x160 [ 9880.066655] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9880.066950] ? do_wait_intr+0xf0/0xf0 [ 9880.066959] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9880.067300] kthread+0x1d1/0x200 [ 9880.067305] ? set_kthread_struct+0x70/0x70 [ 9880.067308] ret_from_fork+0x1f/0x30 [ 9880.074129] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9880.074211] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9880.074218] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9880.074220] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9880.074232] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9880.074236] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9880.074239] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9880.074241] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9880.074243] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9880.074246] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9880.074249] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9880.074252] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9880.074255] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9880.074258] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9880.074268] Call Trace: [ 9880.074274] [ 9880.074280] ? show_regs.cold.9+0x22/0x2f [ 9880.074292] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9880.074298] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9880.074302] ? watchdog+0x40/0x40 [ 9880.074306] ? hrtimer_interrupt+0x128/0x2c0 [ 9880.074310] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9880.408465] ? apic_timer_interrupt+0xf/0x20 [ 9880.412804] [ 9880.416662] ? native_safe_halt+0x2e/0x50 [ 9880.420746] kvm_wait+0xcd/0xe0 [ 9880.424419] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9880.431947] do_raw_spin_lock+0x177/0x190 [ 9880.437646] _raw_spin_lock+0x21/0x30 [ 9880.438841] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9880.443785] ? do_raw_spin_unlock+0x75/0x190 [ 9880.446664] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9880.449570] ? do_wait_intr+0xf0/0xf0 [ 9880.456432] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9880.465494] mdc_reint+0x2d/0x110 [mdc] [ 9880.468611] mdc_unlink+0x658/0x870 [mdc] [ 9880.472329] lmv_unlink+0x34d/0xbd0 [lmv] [ 9880.476953] ll_unlink+0x24d/0x7f0 [lustre] [ 9880.481621] vfs_unlink+0x202/0x330 [ 9880.484540] do_unlinkat+0x2fa/0x460 [ 9880.487660] __x64_sys_unlink+0x37/0x50 [ 9880.490984] do_syscall_64+0xc1/0x440 [ 9880.494287] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9880.496394] RIP: 0033:0x7f83e5f5c10b [ 9880.497563] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9880.509562] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9880.516581] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9880.521125] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9880.524373] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9880.527954] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9880.530653] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9882.083127] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7368s! [ 9882.094451] Showing busy workqueues and worker pools: [ 9882.100575] workqueue events: flags=0x0 [ 9882.103882] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9882.109157] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9882.114072] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9882.116388] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9882.123443] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9882.127421] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9882.130149] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9882.139656] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9882.156924] workqueue events_long: flags=0x0 [ 9882.165116] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9882.168088] pending: lu_objects_flush [obdclass] [ 9882.173067] workqueue events_unbound: flags=0x2 [ 9882.179790] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9882.185713] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9882.192159] workqueue events_power_efficient: flags=0x80 [ 9882.195550] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9882.203181] pending: neigh_periodic_work, do_cache_clean [ 9882.207007] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9882.215549] pending: neigh_periodic_work, check_lifetime [ 9882.223357] workqueue mm_percpu_wq: flags=0x8 [ 9882.229788] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9882.235584] pending: vmstat_update, lru_add_drain_per_cpu [ 9882.240694] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9882.244967] pending: vmstat_update [ 9882.246752] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9882.250666] pending: lru_add_drain_per_cpu BAR(48) [ 9882.252809] workqueue ipv6_addrconf: flags=0x40008 [ 9882.256459] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9882.262446] pending: addrconf_verify_work [ 9882.265915] workqueue knbd0-recv: flags=0x1a [ 9882.270889] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9882.275489] in-flight: 650:recv_work [ 9882.276607] workqueue ptlrpc_pinger: flags=0x2 [ 9882.277819] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9882.285550] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9882.290067] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7369s workers=4 idle: 36 466 29757 [ 9882.300733] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9872s workers=2 idle: 652 [ 9896.060360] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9896.063767] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9896.094710] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9896.101705] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9896.105858] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9896.111459] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9896.132031] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9896.137921] RAX: 00000000000070b7 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9896.142749] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9896.146590] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9896.151327] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9896.154346] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9896.158503] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9896.165728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9896.169701] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9896.184337] Call Trace: [ 9896.185273] [ 9896.186810] ? show_regs.cold.9+0x22/0x2f [ 9896.189476] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9896.192386] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9896.199008] ? watchdog+0x40/0x40 [ 9896.200576] ? hrtimer_interrupt+0x128/0x2c0 [ 9896.203414] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9896.206616] ? apic_timer_interrupt+0xf/0x20 [ 9896.209844] [ 9896.210838] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9896.213694] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9896.219316] do_raw_spin_lock+0x177/0x190 [ 9896.221936] _raw_spin_lock+0x21/0x30 [ 9896.224244] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9896.229358] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9896.231997] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9896.241736] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9896.248424] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9896.252739] ? woken_wake_function+0x30/0x30 [ 9896.254792] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9896.261358] kthread+0x1d1/0x200 [ 9896.264906] ? set_kthread_struct+0x70/0x70 [ 9896.268568] ret_from_fork+0x1f/0x30 [ 9904.060037] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9904.072019] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9904.124594] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9904.144502] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9904.149235] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9904.151628] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9904.157893] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9904.185629] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9904.190448] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9904.192861] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9904.196267] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9904.204092] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9904.212633] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9904.227846] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9904.237501] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9904.241904] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9904.249347] Call Trace: [ 9904.252391] [ 9904.254375] ? show_regs.cold.9+0x22/0x2f [ 9904.262888] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9904.269307] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9904.274231] ? watchdog+0x40/0x40 [ 9904.280122] ? hrtimer_interrupt+0x128/0x2c0 [ 9904.284833] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9904.294399] ? apic_timer_interrupt+0xf/0x20 [ 9904.304699] [ 9904.305504] ? native_safe_halt+0x2e/0x50 [ 9904.311392] kvm_wait+0xcd/0xe0 [ 9904.320803] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9904.324826] do_raw_spin_lock+0x177/0x190 [ 9904.339413] _raw_spin_lock+0x21/0x30 [ 9904.345871] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9904.351787] process_one_work+0x2c8/0x700 [ 9904.354407] worker_thread+0x296/0x6e0 [ 9904.362058] ? rescuer_thread+0x570/0x570 [ 9904.364681] kthread+0x1d1/0x200 [ 9904.370853] ? set_kthread_struct+0x70/0x70 [ 9904.375912] ret_from_fork+0x1f/0x30 [ 9908.059834] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 9908.062594] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 9908.063332] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9908.063405] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9908.063412] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9908.063415] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9908.063428] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9908.063432] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9908.063436] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9908.063439] RDX: 000000000000673f RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9908.063441] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9908.063443] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9908.063446] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9908.063449] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9908.063452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9908.063455] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9908.063463] Call Trace: [ 9908.063469] [ 9908.063474] ? show_regs.cold.9+0x22/0x2f [ 9908.063483] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9908.063489] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9908.063494] ? watchdog+0x40/0x40 [ 9908.063499] ? hrtimer_interrupt+0x128/0x2c0 [ 9908.063503] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9908.063511] ? apic_timer_interrupt+0xf/0x20 [ 9908.063515] [ 9908.063518] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9908.063522] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9908.063526] do_raw_spin_lock+0x177/0x190 [ 9908.063531] _raw_spin_lock+0x21/0x30 [ 9908.063537] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9908.068647] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9908.068719] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9908.068726] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9908.068729] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9908.068743] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9908.068747] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9908.068751] RAX: 00000000000004d7 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9908.068753] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9908.068755] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9908.068757] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9908.068759] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9908.068762] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9908.068765] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9908.068768] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9908.068777] Call Trace: [ 9908.068791] [ 9908.068797] ? show_regs.cold.9+0x22/0x2f [ 9908.068805] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9908.068812] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9908.068816] ? watchdog+0x40/0x40 [ 9908.068820] ? hrtimer_interrupt+0x128/0x2c0 [ 9908.068824] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9908.068973] ? apic_timer_interrupt+0xf/0x20 [ 9908.068981] [ 9908.068985] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9908.068990] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9908.068994] do_raw_spin_lock+0x177/0x190 [ 9908.068998] _raw_spin_lock+0x21/0x30 [ 9908.069004] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9908.069372] ? do_raw_spin_unlock+0x75/0x190 [ 9908.069379] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9908.069578] ? do_wait_intr+0xf0/0xf0 [ 9908.069585] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9908.070763] mdc_reint+0x2d/0x110 [mdc] [ 9908.071147] mdc_unlink+0x658/0x870 [mdc] [ 9908.071177] lmv_unlink+0x34d/0xbd0 [lmv] [ 9908.071206] ll_unlink+0x24d/0x7f0 [lustre] [ 9908.071416] vfs_unlink+0x202/0x330 [ 9908.071427] do_unlinkat+0x2fa/0x460 [ 9908.071432] __x64_sys_unlink+0x37/0x50 [ 9908.071435] do_syscall_64+0xc1/0x440 [ 9908.071548] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9908.071557] RIP: 0033:0x7f83e5f5c10b [ 9908.071563] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9908.071567] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9908.071573] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9908.071575] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9908.071577] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9908.071579] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9908.071581] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9908.469706] obd_mod_free+0x46/0x140 [ptlrpc] [ 9908.473518] mdc_commit_open+0x91/0xb0 [mdc] [ 9908.475083] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9908.476988] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9908.481936] after_reply+0xe2c/0x1780 [ptlrpc] [ 9908.486207] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9908.489861] ? __next_timer_interrupt+0x160/0x160 [ 9908.494163] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9908.499506] ? do_wait_intr+0xf0/0xf0 [ 9908.503807] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9908.506976] kthread+0x1d1/0x200 [ 9908.509150] ? set_kthread_struct+0x70/0x70 [ 9908.512019] ret_from_fork+0x1f/0x30 [ 9912.807927] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7399s! [ 9912.815615] Showing busy workqueues and worker pools: [ 9912.820545] workqueue events: flags=0x0 [ 9912.822465] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9912.825912] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9912.840065] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9912.847888] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9912.851464] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9912.855865] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9912.865874] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9912.872550] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9912.884789] workqueue events_long: flags=0x0 [ 9912.892275] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9912.900775] pending: lu_objects_flush [obdclass] [ 9912.906885] workqueue events_unbound: flags=0x2 [ 9912.913721] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9912.924566] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9912.927985] workqueue events_power_efficient: flags=0x80 [ 9912.931940] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9912.940008] pending: neigh_periodic_work, do_cache_clean [ 9912.953349] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9912.955566] pending: neigh_periodic_work, check_lifetime [ 9912.963778] workqueue mm_percpu_wq: flags=0x8 [ 9912.976811] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9912.984470] pending: vmstat_update, lru_add_drain_per_cpu [ 9912.989510] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9913.004037] pending: vmstat_update [ 9913.005418] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9913.012550] pending: lru_add_drain_per_cpu BAR(48) [ 9913.019182] workqueue ipv6_addrconf: flags=0x40008 [ 9913.020747] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9913.039884] pending: addrconf_verify_work [ 9913.050865] workqueue knbd0-recv: flags=0x1a [ 9913.055965] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9913.062899] in-flight: 650:recv_work [ 9913.068237] workqueue ptlrpc_pinger: flags=0x2 [ 9913.069761] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9913.085277] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9913.099490] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7399s workers=4 idle: 36 466 29757 [ 9913.116176] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9903s workers=2 idle: 652 [ 9924.060131] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9924.064345] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9924.081200] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9924.085267] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9924.088203] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9924.090180] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9924.097279] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9924.100279] RAX: 0000000000006385 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9924.103277] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9924.106392] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9924.109045] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9924.111255] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9924.113573] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9924.116755] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9924.118840] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9924.121623] Call Trace: [ 9924.122637] [ 9924.123490] ? show_regs.cold.9+0x22/0x2f [ 9924.125163] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9924.126985] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9924.128757] ? watchdog+0x40/0x40 [ 9924.130106] ? hrtimer_interrupt+0x128/0x2c0 [ 9924.131353] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9924.132943] ? apic_timer_interrupt+0xf/0x20 [ 9924.134502] [ 9924.135307] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9924.137751] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9924.139716] do_raw_spin_lock+0x177/0x190 [ 9924.141240] _raw_spin_lock+0x21/0x30 [ 9924.142549] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9924.145274] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9924.147514] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9924.150060] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9924.152333] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9924.154482] ? woken_wake_function+0x30/0x30 [ 9924.155986] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9924.158466] kthread+0x1d1/0x200 [ 9924.159798] ? set_kthread_struct+0x70/0x70 [ 9924.161304] ret_from_fork+0x1f/0x30 [ 9932.063427] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [ 9932.088009] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9932.167580] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9932.175667] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9932.186482] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9932.198252] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9932.212394] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9932.238578] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9932.252532] RAX: 0000000000007d0a RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9932.272060] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9932.284005] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9932.293213] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9932.299170] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9932.312969] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9932.324174] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9932.331338] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9932.338180] Call Trace: [ 9932.339208] [ 9932.341136] ? show_regs.cold.9+0x22/0x2f [ 9932.345702] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9932.349240] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9932.351625] ? watchdog+0x40/0x40 [ 9932.353167] ? hrtimer_interrupt+0x128/0x2c0 [ 9932.359945] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9932.363755] ? apic_timer_interrupt+0xf/0x20 [ 9932.366025] [ 9932.368692] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9932.372801] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9932.386800] do_raw_spin_lock+0x177/0x190 [ 9932.391103] _raw_spin_lock+0x21/0x30 [ 9932.394663] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9932.402632] process_one_work+0x2c8/0x700 [ 9932.407397] worker_thread+0x296/0x6e0 [ 9932.408794] ? rescuer_thread+0x570/0x570 [ 9932.412964] kthread+0x1d1/0x200 [ 9932.414110] ? set_kthread_struct+0x70/0x70 [ 9932.420598] ret_from_fork+0x1f/0x30 [ 9936.065674] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [ 9936.079186] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9936.154478] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9936.163254] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9936.180536] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9936.188198] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9936.204065] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9936.222154] RAX: 0000000000003f8f RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9936.231741] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9936.243626] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9936.252307] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9936.264873] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9936.268300] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9936.280708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9936.287526] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9936.298224] Call Trace: [ 9936.300449] [ 9936.301728] ? show_regs.cold.9+0x22/0x2f [ 9936.306061] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9936.313656] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9936.319496] ? watchdog+0x40/0x40 [ 9936.324181] ? hrtimer_interrupt+0x128/0x2c0 [ 9936.330631] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9936.332440] ? apic_timer_interrupt+0xf/0x20 [ 9936.340136] [ 9936.341323] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9936.351956] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9936.358833] do_raw_spin_lock+0x177/0x190 [ 9936.361090] _raw_spin_lock+0x21/0x30 [ 9936.375214] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9936.384515] ? do_raw_spin_unlock+0x75/0x190 [ 9936.396494] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9936.404441] ? do_wait_intr+0xf0/0xf0 [ 9936.405897] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9936.417425] mdc_reint+0x2d/0x110 [mdc] [ 9936.418877] mdc_unlink+0x658/0x870 [mdc] [ 9936.426961] lmv_unlink+0x34d/0xbd0 [lmv] [ 9936.434673] ll_unlink+0x24d/0x7f0 [lustre] [ 9936.443551] vfs_unlink+0x202/0x330 [ 9936.446336] do_unlinkat+0x2fa/0x460 [ 9936.448858] __x64_sys_unlink+0x37/0x50 [ 9936.454290] do_syscall_64+0xc1/0x440 [ 9936.459140] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9936.470291] RIP: 0033:0x7f83e5f5c10b [ 9936.471606] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9936.484533] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9936.493079] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9936.495953] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9936.504474] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9936.512789] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9936.531189] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9940.056060] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [ 9940.060544] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9940.087213] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9940.093225] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9940.098582] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9940.101428] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9940.111551] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9940.115260] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9940.119478] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9940.123510] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9940.127642] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9940.132227] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9940.138713] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9940.142842] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9940.152555] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9940.161434] Call Trace: [ 9940.164251] [ 9940.166668] ? show_regs.cold.9+0x22/0x2f [ 9940.171524] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9940.174472] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9940.178939] ? watchdog+0x40/0x40 [ 9940.181415] ? hrtimer_interrupt+0x128/0x2c0 [ 9940.184908] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9940.188377] ? apic_timer_interrupt+0xf/0x20 [ 9940.190915] [ 9940.192179] ? native_safe_halt+0x2e/0x50 [ 9940.196985] kvm_wait+0xcd/0xe0 [ 9940.199276] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9940.202907] do_raw_spin_lock+0x177/0x190 [ 9940.205602] _raw_spin_lock+0x21/0x30 [ 9940.208539] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9940.211079] obd_mod_free+0x46/0x140 [ptlrpc] [ 9940.213600] mdc_commit_open+0x91/0xb0 [mdc] [ 9940.216872] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9940.220165] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9940.223714] after_reply+0xe2c/0x1780 [ptlrpc] [ 9940.225355] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9940.228062] ? __next_timer_interrupt+0x160/0x160 [ 9940.231936] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9940.235285] ? do_wait_intr+0xf0/0xf0 [ 9940.238205] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9940.239813] kthread+0x1d1/0x200 [ 9940.241098] ? set_kthread_struct+0x70/0x70 [ 9940.242295] ret_from_fork+0x1f/0x30 [ 9943.529450] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7430s! [ 9943.534141] Showing busy workqueues and worker pools: [ 9943.542682] workqueue events: flags=0x0 [ 9943.545142] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9943.548590] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9943.553035] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9943.556776] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9943.560621] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9943.564172] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9943.568484] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9943.571854] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9943.576486] workqueue events_long: flags=0x0 [ 9943.578608] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9943.581822] pending: lu_objects_flush [obdclass] [ 9943.588367] workqueue events_unbound: flags=0x2 [ 9943.590603] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9943.596539] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9943.602021] workqueue events_power_efficient: flags=0x80 [ 9943.605894] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9943.612452] pending: neigh_periodic_work, do_cache_clean [ 9943.618859] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9943.629093] pending: neigh_periodic_work, check_lifetime [ 9943.634275] workqueue mm_percpu_wq: flags=0x8 [ 9943.646959] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9943.654626] pending: vmstat_update, lru_add_drain_per_cpu [ 9943.658416] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9943.666950] pending: vmstat_update [ 9943.669354] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9943.675833] pending: lru_add_drain_per_cpu BAR(48) [ 9943.680547] workqueue ipv6_addrconf: flags=0x40008 [ 9943.683119] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9943.687995] pending: addrconf_verify_work [ 9943.697267] workqueue knbd0-recv: flags=0x1a [ 9943.701488] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9943.706660] in-flight: 650:recv_work [ 9943.710311] workqueue ptlrpc_pinger: flags=0x2 [ 9943.713058] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9943.717145] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9943.720682] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7430s workers=4 idle: 36 466 29757 [ 9943.725465] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9933s workers=2 idle: 652 [ 9952.059025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [ 9952.062257] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9952.101604] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9952.111456] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9952.117540] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9952.123190] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9952.139135] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9952.146169] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9952.156563] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9952.160002] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9952.166138] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9952.174134] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9952.179220] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9952.189722] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9952.195938] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9952.203710] Call Trace: [ 9952.207424] [ 9952.210410] ? show_regs.cold.9+0x22/0x2f [ 9952.214996] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9952.216720] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9952.218343] ? watchdog+0x40/0x40 [ 9952.227366] ? hrtimer_interrupt+0x128/0x2c0 [ 9952.231443] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9952.239082] ? apic_timer_interrupt+0xf/0x20 [ 9952.246203] [ 9952.247045] ? native_safe_halt+0x2e/0x50 [ 9952.251279] kvm_wait+0xcd/0xe0 [ 9952.254342] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9952.261185] do_raw_spin_lock+0x177/0x190 [ 9952.265940] _raw_spin_lock+0x21/0x30 [ 9952.268291] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9952.278607] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9952.290512] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9952.295627] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9952.308569] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9952.318991] ? woken_wake_function+0x30/0x30 [ 9952.324884] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9952.330886] kthread+0x1d1/0x200 [ 9952.334427] ? set_kthread_struct+0x70/0x70 [ 9952.339283] ret_from_fork+0x1f/0x30 [ 9960.065366] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 9960.076451] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9960.157204] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9960.178892] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9960.193736] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9960.204829] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9960.213771] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9960.242676] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9960.252370] RAX: 0000000000007282 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9960.265867] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9960.276156] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9960.290867] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9960.303174] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9960.310503] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9960.325200] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9960.332769] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9960.345348] Call Trace: [ 9960.346357] [ 9960.347550] ? show_regs.cold.9+0x22/0x2f [ 9960.354288] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9960.364577] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9960.374065] ? watchdog+0x40/0x40 [ 9960.384590] ? hrtimer_interrupt+0x128/0x2c0 [ 9960.391874] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9960.403486] ? apic_timer_interrupt+0xf/0x20 [ 9960.410511] [ 9960.414944] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9960.423987] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9960.432642] do_raw_spin_lock+0x177/0x190 [ 9960.441875] _raw_spin_lock+0x21/0x30 [ 9960.453483] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9960.463887] process_one_work+0x2c8/0x700 [ 9960.472354] worker_thread+0x296/0x6e0 [ 9960.476103] ? rescuer_thread+0x570/0x570 [ 9960.484811] kthread+0x1d1/0x200 [ 9960.488427] ? set_kthread_struct+0x70/0x70 [ 9960.493685] ret_from_fork+0x1f/0x30 [ 9961.154885] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 9961.160713] rcu: 0-....: (5015686 ticks this GP) idle=30a/1/0x4000000000000000 softirq=128519/128519 fqs=126395 [ 9961.166871] (detected by 1, t=7980195 jiffies, g=262441, q=59639) [ 9961.170441] Sending NMI from CPU 1 to CPUs 0: [ 9961.179493] NMI backtrace for cpu 0 [ 9961.179501] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9961.179512] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9961.179513] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9961.179515] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9961.179517] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 [ 9961.179520] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [ 9961.179522] RDX: 0000000000007d1c RSI: 0000000000000003 RDI: ffff9a61907912e0 [ 9961.179523] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [ 9961.179525] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [ 9961.179526] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9961.179527] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [ 9961.179529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9961.179531] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [ 9961.179533] Call Trace: [ 9961.179534] [ 9961.179535] ? show_regs.cold.9+0x22/0x2f [ 9961.179536] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9961.179537] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9961.179539] ? nmi_handle+0xbd/0x210 [ 9961.179540] ? default_do_nmi+0x4b/0x1b0 [ 9961.179541] ? do_nmi+0x280/0x380 [ 9961.179543] ? end_repeat_nmi+0x16/0x1f [ 9961.179544] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9961.179545] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9961.179546] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9961.179548] [ 9961.179549] do_raw_spin_lock+0x177/0x190 [ 9961.179550] _raw_spin_lock+0x21/0x30 [ 9961.179551] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9961.179552] obd_mod_free+0x46/0x140 [ptlrpc] [ 9961.179554] mdc_commit_open+0x91/0xb0 [mdc] [ 9961.179555] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9961.179556] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9961.179558] after_reply+0xe2c/0x1780 [ptlrpc] [ 9961.179559] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9961.179560] ? __next_timer_interrupt+0x160/0x160 [ 9961.179561] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9961.179562] ? do_wait_intr+0xf0/0xf0 [ 9961.179563] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9961.179564] kthread+0x1d1/0x200 [ 9961.179566] ? set_kthread_struct+0x70/0x70 [ 9961.179567] ret_from_fork+0x1f/0x30 [ 9961.183408] rcu: rcu_sched kthread starved for 7449382 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 9961.440752] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9961.449104] rcu: RCU grace-period kthread stack dump: [ 9961.458161] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9961.473991] Call Trace: [ 9961.486539] __schedule+0x351/0xcb0 [ 9961.495060] schedule+0xc0/0x180 [ 9961.501469] schedule_timeout+0xb4/0x190 [ 9961.511006] ? do_raw_spin_unlock+0x75/0x190 [ 9961.516316] ? __next_timer_interrupt+0x160/0x160 [ 9961.532566] ? prepare_to_swait_event+0xf3/0x140 [ 9961.534157] rcu_gp_kthread+0x8dd/0x15b0 [ 9961.542070] ? force_qs_rnp+0x2c0/0x2c0 [ 9961.547022] kthread+0x1d1/0x200 [ 9961.549148] ? set_kthread_struct+0x70/0x70 [ 9961.555637] ret_from_fork+0x1f/0x30 [ 9961.560035] rcu: Stack dump where RCU GP kthread last ran: [ 9961.567211] Sending NMI from CPU 1 to CPUs 3: [ 9961.570752] NMI backtrace for cpu 3 [ 9961.570760] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9961.570762] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9961.570764] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9961.570766] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9961.570768] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 [ 9961.570770] RAX: 000000000000498d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9961.570772] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9961.570773] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9961.570775] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9961.570777] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9961.570778] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9961.570780] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9961.570781] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9961.570782] Call Trace: [ 9961.570784] [ 9961.570784] ? show_regs.cold.9+0x22/0x2f [ 9961.570786] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9961.570787] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9961.570788] ? nmi_handle+0xbd/0x210 [ 9961.570789] ? default_do_nmi+0x4b/0x1b0 [ 9961.570790] ? do_nmi+0x280/0x380 [ 9961.570791] ? end_repeat_nmi+0x16/0x1f [ 9961.570792] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9961.570794] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9961.570795] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9961.570796] [ 9961.570797] do_raw_spin_lock+0x177/0x190 [ 9961.570798] _raw_spin_lock+0x21/0x30 [ 9961.570799] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9961.570800] ? do_raw_spin_unlock+0x75/0x190 [ 9961.570801] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9961.570803] ? do_wait_intr+0xf0/0xf0 [ 9961.570804] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9961.570806] mdc_reint+0x2d/0x110 [mdc] [ 9961.570807] mdc_unlink+0x658/0x870 [mdc] [ 9961.570808] lmv_unlink+0x34d/0xbd0 [lmv] [ 9961.570809] ll_unlink+0x24d/0x7f0 [lustre] [ 9961.570810] vfs_unlink+0x202/0x330 [ 9961.570811] do_unlinkat+0x2fa/0x460 [ 9961.570813] __x64_sys_unlink+0x37/0x50 [ 9961.570814] do_syscall_64+0xc1/0x440 [ 9961.570815] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9961.570816] RIP: 0033:0x7f83e5f5c10b [ 9961.570817] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9961.570820] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9961.570823] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9961.570825] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9961.570826] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9961.570828] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9961.570830] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9973.728582] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7460s! [ 9973.734639] Showing busy workqueues and worker pools: [ 9973.740988] workqueue events: flags=0x0 [ 9973.745034] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9973.748963] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [ 9973.754086] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9973.763123] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [ 9973.775992] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9973.782763] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [ 9973.786247] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9973.789737] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [ 9973.793815] workqueue events_long: flags=0x0 [ 9973.795509] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9973.798584] pending: lu_objects_flush [obdclass] [ 9973.801046] workqueue events_unbound: flags=0x2 [ 9973.802646] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9973.805334] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9973.808030] workqueue events_power_efficient: flags=0x80 [ 9973.810125] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9973.812809] pending: neigh_periodic_work, do_cache_clean [ 9973.815022] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9973.819968] pending: neigh_periodic_work, check_lifetime [ 9973.822369] workqueue mm_percpu_wq: flags=0x8 [ 9973.823945] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9973.826766] pending: vmstat_update, lru_add_drain_per_cpu [ 9973.829195] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9973.831392] pending: vmstat_update [ 9973.832601] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [ 9973.834787] pending: lru_add_drain_per_cpu BAR(48) [ 9973.836392] workqueue ipv6_addrconf: flags=0x40008 [ 9973.837698] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9973.840696] pending: addrconf_verify_work [ 9973.842728] workqueue knbd0-recv: flags=0x1a [ 9973.844560] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9973.847980] in-flight: 650:recv_work [ 9973.849895] workqueue ptlrpc_pinger: flags=0x2 [ 9973.851841] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9973.853979] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [ 9973.856311] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7460s workers=4 idle: 36 466 29757 [ 9973.859530] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9963s workers=2 idle: 652 [ 9988.059086] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [ 9988.060076] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [ 9988.062013] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [ 9988.062032] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9988.062118] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9988.062124] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9988.062127] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9988.062141] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9988.062145] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9988.062149] RAX: 0000000000007a4d RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9988.062152] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [ 9988.062154] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9988.062157] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [ 9988.062159] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [ 9988.062163] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [ 9988.062166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9988.062169] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [ 9988.062178] Call Trace: [ 9988.062184] [ 9988.062191] ? show_regs.cold.9+0x22/0x2f [ 9988.062202] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9988.062222] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9988.062226] ? watchdog+0x40/0x40 [ 9988.062230] ? hrtimer_interrupt+0x128/0x2c0 [ 9988.062234] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9988.062243] ? apic_timer_interrupt+0xf/0x20 [ 9988.062248] [ 9988.062250] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9988.062255] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9988.062259] do_raw_spin_lock+0x177/0x190 [ 9988.062264] _raw_spin_lock+0x21/0x30 [ 9988.062270] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9988.062670] ? do_raw_spin_unlock+0x75/0x190 [ 9988.062676] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9988.064527] ? do_wait_intr+0xf0/0xf0 [ 9988.064540] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9988.064747] mdc_reint+0x2d/0x110 [mdc] [ 9988.064794] mdc_unlink+0x658/0x870 [mdc] [ 9988.064821] lmv_unlink+0x34d/0xbd0 [lmv] [ 9988.064854] ll_unlink+0x24d/0x7f0 [lustre] [ 9988.064989] vfs_unlink+0x202/0x330 [ 9988.064999] do_unlinkat+0x2fa/0x460 [ 9988.065003] __x64_sys_unlink+0x37/0x50 [ 9988.065006] do_syscall_64+0xc1/0x440 [ 9988.065015] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9988.065026] RIP: 0033:0x7f83e5f5c10b [ 9988.065035] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [ 9988.065040] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 9988.065046] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [ 9988.065048] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [ 9988.065052] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [ 9988.065054] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [ 9988.065057] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 9988.070249] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9988.070550] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9988.070559] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9988.070562] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9988.070578] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9988.070582] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9988.070588] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9988.070590] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [ 9988.070592] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9988.070594] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [ 9988.070597] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [ 9988.070600] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [ 9988.070603] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9988.070605] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [ 9988.070615] Call Trace: [ 9988.070623] [ 9988.070631] ? show_regs.cold.9+0x22/0x2f [ 9988.070645] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9988.070652] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9988.070656] ? watchdog+0x40/0x40 [ 9988.070661] ? hrtimer_interrupt+0x128/0x2c0 [ 9988.070666] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9988.070674] ? apic_timer_interrupt+0xf/0x20 [ 9988.070774] [ 9988.070778] ? native_safe_halt+0x2e/0x50 [ 9988.070783] kvm_wait+0xcd/0xe0 [ 9988.070790] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9988.070798] do_raw_spin_lock+0x177/0x190 [ 9988.070803] _raw_spin_lock+0x21/0x30 [ 9988.070807] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9988.079509] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9988.079588] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9988.079595] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9988.079603] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9988.095666] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9988.155906] [ 9988.155916] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9988.155943] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9988.155947] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9988.155952] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [ 9988.155955] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [ 9988.155957] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [ 9988.156049] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [ 9988.156051] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [ 9988.156055] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [ 9988.156058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9988.156061] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [ 9988.156078] Call Trace: [ 9988.156084] [ 9988.156091] ? show_regs.cold.9+0x22/0x2f [ 9988.156102] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9988.156110] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9988.156114] ? watchdog+0x40/0x40 [ 9988.156119] ? hrtimer_interrupt+0x128/0x2c0 [ 9988.156123] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9988.156130] ? apic_timer_interrupt+0xf/0x20 [ 9988.156134] [ 9988.156138] ? native_safe_halt+0x2e/0x50 [ 9988.156143] kvm_wait+0xcd/0xe0 [ 9988.156151] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9988.156163] do_raw_spin_lock+0x177/0x190 [ 9988.156168] _raw_spin_lock+0x21/0x30 [ 9988.156173] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9988.157160] process_one_work+0x2c8/0x700 [ 9988.157169] worker_thread+0x296/0x6e0 [ 9988.157174] ? rescuer_thread+0x570/0x570 [ 9988.157179] kthread+0x1d1/0x200 [ 9988.157183] ? set_kthread_struct+0x70/0x70 [ 9988.157187] ret_from_fork+0x1f/0x30 [ 9989.044752] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9989.049130] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9989.052701] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9989.055500] ? woken_wake_function+0x30/0x30 [ 9989.057528] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9989.060441] kthread+0x1d1/0x200 [ 9989.061544] ? set_kthread_struct+0x70/0x70 [ 9989.063006] ret_from_fork+0x1f/0x30 [10000.066705] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [10000.083119] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10000.115653] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10000.123477] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10000.128363] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10000.135081] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10000.147653] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10000.155422] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10000.159391] RDX: 0000000000004360 RSI: 0000000000000003 RDI: ffff9a61907912e0 [10000.163070] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10000.166497] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10000.172105] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10000.174545] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10000.177595] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10000.182656] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10000.189098] Call Trace: [10000.190084] [10000.190685] ? show_regs.cold.9+0x22/0x2f [10000.192542] ? watchdog_timer_fn.cold.6+0x62/0x10a [10000.194894] ? __hrtimer_run_queues+0x1c2/0x4d0 [10000.196875] ? watchdog+0x40/0x40 [10000.198773] ? hrtimer_interrupt+0x128/0x2c0 [10000.205600] ? smp_apic_timer_interrupt+0xa6/0x270 [10000.211445] ? apic_timer_interrupt+0xf/0x20 [10000.213834] [10000.214843] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10000.226450] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10000.228608] do_raw_spin_lock+0x177/0x190 [10000.236699] _raw_spin_lock+0x21/0x30 [10000.238085] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10000.248736] obd_mod_free+0x46/0x140 [ptlrpc] [10000.256086] mdc_commit_open+0x91/0xb0 [mdc] [10000.258669] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10000.261594] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10000.265678] after_reply+0xe2c/0x1780 [ptlrpc] [10000.269828] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10000.278392] ? __next_timer_interrupt+0x160/0x160 [10000.281084] ptlrpcd+0xa82/0xe40 [ptlrpc] [10000.283333] ? do_wait_intr+0xf0/0xf0 [10000.284612] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10000.287608] kthread+0x1d1/0x200 [10000.289134] ? set_kthread_struct+0x70/0x70 [10000.291439] ret_from_fork+0x1f/0x30 [10003.937130] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7490s! [10003.942500] Showing busy workqueues and worker pools: [10003.945774] workqueue events: flags=0x0 [10003.949891] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10003.954650] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10003.960765] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10003.965420] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10003.972098] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10003.976063] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10003.984732] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10003.988539] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10003.994180] workqueue events_long: flags=0x0 [10003.996521] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10003.999673] pending: lu_objects_flush [obdclass] [10004.003643] workqueue events_unbound: flags=0x2 [10004.006524] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10004.009575] pending: flush_memcg_stats_dwork, idle_cull_fn [10004.013099] workqueue events_power_efficient: flags=0x80 [10004.016412] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10004.018805] pending: neigh_periodic_work, do_cache_clean [10004.022030] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10004.025617] pending: neigh_periodic_work, check_lifetime [10004.028935] workqueue mm_percpu_wq: flags=0x8 [10004.031332] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10004.037319] pending: vmstat_update, lru_add_drain_per_cpu [10004.042350] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10004.049521] pending: vmstat_update [10004.054876] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10004.064251] pending: lru_add_drain_per_cpu BAR(48) [10004.069658] workqueue ipv6_addrconf: flags=0x40008 [10004.072140] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10004.075619] pending: addrconf_verify_work [10004.077236] workqueue knbd0-recv: flags=0x1a [10004.079874] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10004.083379] in-flight: 650:recv_work [10004.085479] workqueue ptlrpc_pinger: flags=0x2 [10004.087837] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10004.090687] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10004.094335] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7490s workers=4 idle: 36 466 29757 [10004.098584] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9994s workers=2 idle: 652 [10016.059012] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [10016.060026] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [10016.062023] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [10016.062036] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10016.062115] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10016.062121] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10016.062124] RIP: 0010:native_safe_halt+0x2e/0x50 [10016.062139] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10016.062143] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10016.062147] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10016.062149] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10016.062151] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10016.062153] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10016.062155] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10016.062157] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10016.062160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10016.062162] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10016.062171] Call Trace: [10016.062176] [10016.062279] ? show_regs.cold.9+0x22/0x2f [10016.062292] ? watchdog_timer_fn.cold.6+0x62/0x10a [10016.062300] ? __hrtimer_run_queues+0x1c2/0x4d0 [10016.062307] ? watchdog+0x40/0x40 [10016.062314] ? hrtimer_interrupt+0x128/0x2c0 [10016.062320] ? smp_apic_timer_interrupt+0xa6/0x270 [10016.062334] ? apic_timer_interrupt+0xf/0x20 [10016.062344] [10016.062348] ? native_safe_halt+0x2e/0x50 [10016.062355] kvm_wait+0xcd/0xe0 [10016.062667] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10016.062675] do_raw_spin_lock+0x177/0x190 [10016.062680] _raw_spin_lock+0x21/0x30 [10016.062686] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10016.063020] ? do_raw_spin_unlock+0x75/0x190 [10016.063141] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) [10016.063026] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10016.063163] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10016.063205] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10016.063211] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10016.063214] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10016.063223] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10016.063226] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10016.063231] RAX: 0000000000003d36 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10016.063233] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10016.063235] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10016.063237] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10016.063239] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10016.063241] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10016.063244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10016.063246] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10016.063252] Call Trace: [10016.063255] [10016.063258] ? show_regs.cold.9+0x22/0x2f [10016.063266] ? watchdog_timer_fn.cold.6+0x62/0x10a [10016.063271] ? __hrtimer_run_queues+0x1c2/0x4d0 [10016.063274] ? watchdog+0x40/0x40 [10016.063278] ? hrtimer_interrupt+0x128/0x2c0 [10016.063282] ? smp_apic_timer_interrupt+0xa6/0x270 [10016.063288] ? apic_timer_interrupt+0xf/0x20 [10016.063292] [10016.063294] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10016.063299] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10016.063303] do_raw_spin_lock+0x177/0x190 [10016.063307] _raw_spin_lock+0x21/0x30 [10016.063369] ? do_wait_intr+0xf0/0xf0 [10016.063311] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10016.063378] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10016.064422] mdc_reint+0x2d/0x110 [mdc] [10016.064499] mdc_unlink+0x658/0x870 [mdc] [10016.064684] lmv_unlink+0x34d/0xbd0 [lmv] [10016.064751] ll_unlink+0x24d/0x7f0 [lustre] [10016.064916] vfs_unlink+0x202/0x330 [10016.064924] do_unlinkat+0x2fa/0x460 [10016.064930] __x64_sys_unlink+0x37/0x50 [10016.064933] do_syscall_64+0xc1/0x440 [10016.064940] entry_SYSCALL_64_after_hwframe+0x49/0xae [10016.064947] RIP: 0033:0x7f83e5f5c10b [10016.064954] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10016.064958] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10016.064962] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10016.064964] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10016.064966] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10016.064968] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10016.064970] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10016.067813] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10016.067895] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10016.067908] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10016.067915] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10016.068479] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10016.068491] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10016.068495] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10016.068499] RAX: 0000000000004977 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10016.068501] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10016.068503] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10016.068505] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10016.068508] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10016.068515] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10016.068519] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10016.068521] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10016.068528] Call Trace: [10016.068533] [10016.068539] ? show_regs.cold.9+0x22/0x2f [10016.068554] ? watchdog_timer_fn.cold.6+0x62/0x10a [10016.068560] ? __hrtimer_run_queues+0x1c2/0x4d0 [10016.068564] ? watchdog+0x40/0x40 [10016.068568] ? hrtimer_interrupt+0x128/0x2c0 [10016.068572] ? smp_apic_timer_interrupt+0xa6/0x270 [10016.068580] ? apic_timer_interrupt+0xf/0x20 [10016.068585] [10016.068588] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10016.068646] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10016.068658] do_raw_spin_lock+0x177/0x190 [10016.068666] _raw_spin_lock+0x21/0x30 [10016.068671] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10016.072454] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10016.072944] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10016.073141] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10016.073318] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10016.073515] ? woken_wake_function+0x30/0x30 [10016.073527] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10016.073730] kthread+0x1d1/0x200 [10016.073739] ? set_kthread_struct+0x70/0x70 [10016.073743] ret_from_fork+0x1f/0x30 [10016.676934] process_one_work+0x2c8/0x700 [10016.680565] worker_thread+0x296/0x6e0 [10016.683808] ? rescuer_thread+0x570/0x570 [10016.688241] kthread+0x1d1/0x200 [10016.690765] ? set_kthread_struct+0x70/0x70 [10016.699440] ret_from_fork+0x1f/0x30 [10032.056005] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [10032.074602] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10032.134397] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10032.155466] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10032.168405] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10032.172424] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10032.187368] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10032.191329] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10032.201527] RDX: 0000000000007f69 RSI: 0000000000000003 RDI: ffff9a61907912e0 [10032.209742] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10032.218172] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10032.229713] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10032.243287] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10032.252481] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10032.257694] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10032.268471] Call Trace: [10032.269678] [10032.270792] ? show_regs.cold.9+0x22/0x2f [10032.272408] ? watchdog_timer_fn.cold.6+0x62/0x10a [10032.275746] ? __hrtimer_run_queues+0x1c2/0x4d0 [10032.277496] ? watchdog+0x40/0x40 [10032.281533] ? hrtimer_interrupt+0x128/0x2c0 [10032.284012] ? smp_apic_timer_interrupt+0xa6/0x270 [10032.286458] ? apic_timer_interrupt+0xf/0x20 [10032.289150] [10032.289870] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10032.291711] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10032.296507] do_raw_spin_lock+0x177/0x190 [10032.298174] _raw_spin_lock+0x21/0x30 [10032.301621] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10032.306353] obd_mod_free+0x46/0x140 [ptlrpc] [10032.311210] mdc_commit_open+0x91/0xb0 [mdc] [10032.313273] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10032.317518] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10032.326739] after_reply+0xe2c/0x1780 [ptlrpc] [10032.329481] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10032.337228] ? __next_timer_interrupt+0x160/0x160 [10032.341352] ptlrpcd+0xa82/0xe40 [ptlrpc] [10032.346448] ? do_wait_intr+0xf0/0xf0 [10032.349293] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10032.353443] kthread+0x1d1/0x200 [10032.356432] ? set_kthread_struct+0x70/0x70 [10032.360528] ret_from_fork+0x1f/0x30 [10034.152218] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7520s! [10034.158014] Showing busy workqueues and worker pools: [10034.162416] workqueue events: flags=0x0 [10034.165539] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10034.170673] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10034.178537] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10034.184187] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10034.188276] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10034.191935] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10034.196496] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10034.208430] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10034.214428] workqueue events_long: flags=0x0 [10034.217786] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10034.223398] pending: lu_objects_flush [obdclass] [10034.225686] workqueue events_unbound: flags=0x2 [10034.227673] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10034.231754] pending: flush_memcg_stats_dwork, idle_cull_fn [10034.236399] workqueue events_power_efficient: flags=0x80 [10034.239679] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10034.243701] pending: neigh_periodic_work, do_cache_clean [10034.246398] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10034.250384] pending: neigh_periodic_work, check_lifetime [10034.254017] workqueue mm_percpu_wq: flags=0x8 [10034.257240] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10034.259312] pending: vmstat_update, lru_add_drain_per_cpu [10034.262218] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10034.267353] pending: vmstat_update [10034.268694] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10034.270680] pending: lru_add_drain_per_cpu BAR(48) [10034.273632] workqueue ipv6_addrconf: flags=0x40008 [10034.275587] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10034.279776] pending: addrconf_verify_work [10034.283247] workqueue knbd0-recv: flags=0x1a [10034.285012] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10034.287889] in-flight: 650:recv_work [10034.289738] workqueue ptlrpc_pinger: flags=0x2 [10034.293589] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10034.297738] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10034.300389] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7521s workers=4 idle: 36 466 29757 [10034.305341] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10024s workers=2 idle: 652 [10044.059402] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [10044.062287] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [10044.063330] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10044.063402] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10044.063410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10044.063413] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10044.063427] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10044.063431] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10044.063436] RAX: 0000000000002510 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10044.063439] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10044.063441] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10044.063443] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10044.063446] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10044.063449] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10044.063452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10044.063455] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10044.063464] Call Trace: [10044.063470] [10044.063476] ? show_regs.cold.9+0x22/0x2f [10044.063487] ? watchdog_timer_fn.cold.6+0x62/0x10a [10044.063493] ? __hrtimer_run_queues+0x1c2/0x4d0 [10044.063498] ? watchdog+0x40/0x40 [10044.063502] ? hrtimer_interrupt+0x128/0x2c0 [10044.063506] ? smp_apic_timer_interrupt+0xa6/0x270 [10044.063514] ? apic_timer_interrupt+0xf/0x20 [10044.063519] [10044.063998] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10044.064005] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10044.064011] do_raw_spin_lock+0x177/0x190 [10044.064017] _raw_spin_lock+0x21/0x30 [10044.064023] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10044.064302] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10044.064494] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10044.064763] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10044.065152] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [10044.065159] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log [10044.065113] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10044.065228] dm_mod sha512_ssse3 sha512_generic [10044.065235] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10044.065241] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10044.065960] ? woken_wake_function+0x30/0x30 [10044.065247] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10044.065971] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10044.066561] [10044.066565] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10044.066574] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10044.066578] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10044.066583] RAX: 0000000000005d86 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10044.066586] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10044.066588] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10044.066590] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10044.066592] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10044.066596] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10044.066600] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10044.066605] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10044.066605] kthread+0x1d1/0x200 [10044.066616] Call Trace: [10044.066615] ? set_kthread_struct+0x70/0x70 [10044.066622] [10044.066624] ret_from_fork+0x1f/0x30 [10044.066629] ? show_regs.cold.9+0x22/0x2f [10044.066642] ? watchdog_timer_fn.cold.6+0x62/0x10a [10044.066650] ? __hrtimer_run_queues+0x1c2/0x4d0 [10044.066654] ? watchdog+0x40/0x40 [10044.066659] ? hrtimer_interrupt+0x128/0x2c0 [10044.066663] ? smp_apic_timer_interrupt+0xa6/0x270 [10044.066670] ? apic_timer_interrupt+0xf/0x20 [10044.066675] [10044.066678] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10044.066684] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10044.066689] do_raw_spin_lock+0x177/0x190 [10044.066694] _raw_spin_lock+0x21/0x30 [10044.066699] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10044.068222] process_one_work+0x2c8/0x700 [10044.068230] worker_thread+0x296/0x6e0 [10044.068236] ? rescuer_thread+0x570/0x570 [10044.068240] kthread+0x1d1/0x200 [10044.068244] ? set_kthread_struct+0x70/0x70 [10044.068248] ret_from_fork+0x1f/0x30 [10044.442210] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10044.482163] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10044.501061] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10044.519168] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10044.524174] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10044.540869] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10044.548306] RAX: 0000000000007c83 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10044.555543] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10044.561363] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10044.578828] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10044.581149] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10044.590269] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10044.592403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10044.600210] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10044.603577] Call Trace: [10044.604668] [10044.605931] ? show_regs.cold.9+0x22/0x2f [10044.609254] ? watchdog_timer_fn.cold.6+0x62/0x10a [10044.626881] ? __hrtimer_run_queues+0x1c2/0x4d0 [10044.633025] ? watchdog+0x40/0x40 [10044.638074] ? hrtimer_interrupt+0x128/0x2c0 [10044.642363] ? smp_apic_timer_interrupt+0xa6/0x270 [10044.646060] ? apic_timer_interrupt+0xf/0x20 [10044.648577] [10044.650587] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10044.657443] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10044.662692] do_raw_spin_lock+0x177/0x190 [10044.668299] _raw_spin_lock+0x21/0x30 [10044.671244] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10044.675174] ? do_raw_spin_unlock+0x75/0x190 [10044.678090] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10044.685207] ? do_wait_intr+0xf0/0xf0 [10044.689363] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10044.697324] mdc_reint+0x2d/0x110 [mdc] [10044.701649] mdc_unlink+0x658/0x870 [mdc] [10044.707948] lmv_unlink+0x34d/0xbd0 [lmv] [10044.710811] ll_unlink+0x24d/0x7f0 [lustre] [10044.722098] vfs_unlink+0x202/0x330 [10044.724459] do_unlinkat+0x2fa/0x460 [10044.727677] __x64_sys_unlink+0x37/0x50 [10044.735969] do_syscall_64+0xc1/0x440 [10044.737119] entry_SYSCALL_64_after_hwframe+0x49/0xae [10044.746174] RIP: 0033:0x7f83e5f5c10b [10044.751560] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10044.761177] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10044.764284] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10044.772079] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10044.780895] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10044.790046] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10044.791954] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10060.059547] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [10060.066246] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10060.106062] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10060.112234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10060.117149] RIP: 0010:native_safe_halt+0x2e/0x50 [10060.122408] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10060.132387] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10060.137861] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10060.141890] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [10060.148831] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10060.152998] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10060.158392] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10060.162283] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10060.165901] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10060.168754] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10060.174426] Call Trace: [10060.175767] [10060.177255] ? show_regs.cold.9+0x22/0x2f [10060.179284] ? watchdog_timer_fn.cold.6+0x62/0x10a [10060.182811] ? __hrtimer_run_queues+0x1c2/0x4d0 [10060.185139] ? watchdog+0x40/0x40 [10060.186304] ? hrtimer_interrupt+0x128/0x2c0 [10060.188819] ? smp_apic_timer_interrupt+0xa6/0x270 [10060.191246] ? apic_timer_interrupt+0xf/0x20 [10060.193120] [10060.194792] ? native_safe_halt+0x2e/0x50 [10060.196616] kvm_wait+0xcd/0xe0 [10060.198162] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10060.200652] do_raw_spin_lock+0x177/0x190 [10060.202230] _raw_spin_lock+0x21/0x30 [10060.204376] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10060.207263] obd_mod_free+0x46/0x140 [ptlrpc] [10060.209220] mdc_commit_open+0x91/0xb0 [mdc] [10060.210737] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10060.214425] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10060.216981] after_reply+0xe2c/0x1780 [ptlrpc] [10060.219062] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10060.221420] ? __next_timer_interrupt+0x160/0x160 [10060.222943] ptlrpcd+0xa82/0xe40 [ptlrpc] [10060.226954] ? do_wait_intr+0xf0/0xf0 [10060.228764] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10060.233025] kthread+0x1d1/0x200 [10060.234218] ? set_kthread_struct+0x70/0x70 [10060.236130] ret_from_fork+0x1f/0x30 [10064.356042] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7551s! [10064.361294] Showing busy workqueues and worker pools: [10064.363767] workqueue events: flags=0x0 [10064.366165] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10064.369025] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10064.373596] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10064.376791] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10064.379984] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10064.383139] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10064.387201] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10064.390543] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10064.395627] workqueue events_long: flags=0x0 [10064.397848] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10064.400937] pending: lu_objects_flush [obdclass] [10064.403672] workqueue events_unbound: flags=0x2 [10064.405404] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10064.407605] pending: flush_memcg_stats_dwork, idle_cull_fn [10064.410026] workqueue events_power_efficient: flags=0x80 [10064.412507] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10064.415656] pending: neigh_periodic_work, do_cache_clean [10064.418239] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10064.421520] pending: neigh_periodic_work, check_lifetime [10064.424235] workqueue mm_percpu_wq: flags=0x8 [10064.426785] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10064.430077] pending: vmstat_update, lru_add_drain_per_cpu [10064.432832] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10064.435578] pending: vmstat_update [10064.437484] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10064.440691] pending: lru_add_drain_per_cpu BAR(48) [10064.442853] workqueue ipv6_addrconf: flags=0x40008 [10064.445079] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10064.448039] pending: addrconf_verify_work [10064.450690] workqueue knbd0-recv: flags=0x1a [10064.452711] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10064.455693] in-flight: 650:recv_work [10064.457623] workqueue ptlrpc_pinger: flags=0x2 [10064.459522] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10064.462406] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10064.464776] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7551s workers=4 idle: 36 466 29757 [10064.468074] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10054s workers=2 idle: 652 [10072.059647] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [10072.062605] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [10072.062930] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [10072.062938] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10072.063011] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10072.063017] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10072.063023] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10072.064939] Modules linked in: [10072.065045] [10072.065054] lustre(O) [10072.065049] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10072.065060] osp(O) ofd(O) lod(O) [10072.065068] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10072.065071] mdt(O) mdd(O) [10072.065077] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 [10072.065079] mgs(O) [10072.065083] ORIG_RAX: ffffffffffffff13 [10072.065084] lquota(O) [10072.065088] RAX: 0000000000007d7a RBX: ffff9a61907912e0 RCX: 0000000000000001 [10072.065089] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) [10072.065105] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10072.065109] ptlrpc_gss(O) [10072.065112] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10072.065114] ptlrpc(O) [10072.065118] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10072.065119] obdclass(O) ksocklnd(O) [10072.065125] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10072.065127] lnet(O) libcfs(O) [10072.065132] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10072.065135] rpcsec_gss_krb5 [10072.065139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10072.065140] auth_rpcgss nfsv4 [10072.065146] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10072.065148] dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 [10072.065158] Call Trace: [10072.065162] squashfs ata_generic crct10dif_pclmul [10072.065167] [10072.065177] ? show_regs.cold.9+0x22/0x2f [10072.065186] ? watchdog_timer_fn.cold.6+0x62/0x10a [10072.065191] ? __hrtimer_run_queues+0x1c2/0x4d0 [10072.065195] ? watchdog+0x40/0x40 [10072.065200] ? hrtimer_interrupt+0x128/0x2c0 [10072.065203] ? smp_apic_timer_interrupt+0xa6/0x270 [10072.065212] ? apic_timer_interrupt+0xf/0x20 [10072.065216] [10072.065218] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10072.065223] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10072.065228] do_raw_spin_lock+0x177/0x190 [10072.065232] crc32_pclmul crc32c_intel [10072.065235] _raw_spin_lock+0x21/0x30 [10072.065240] ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10072.065290] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10072.065296] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10072.065298] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10072.065306] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10072.065310] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10072.065314] RAX: 0000000000000a83 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10072.065316] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10072.065319] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10072.065321] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10072.065324] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10072.065327] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10072.065330] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10072.065332] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10072.065338] Call Trace: [10072.065341] [10072.065344] ? show_regs.cold.9+0x22/0x2f [10072.065350] ? watchdog_timer_fn.cold.6+0x62/0x10a [10072.065355] ? __hrtimer_run_queues+0x1c2/0x4d0 [10072.065359] ? watchdog+0x40/0x40 [10072.065247] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10072.065365] ? hrtimer_interrupt+0x128/0x2c0 [10072.065370] ? smp_apic_timer_interrupt+0xa6/0x270 [10072.065376] ? apic_timer_interrupt+0xf/0x20 [10072.065381] [10072.065384] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10072.065388] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10072.065394] do_raw_spin_lock+0x177/0x190 [10072.065398] _raw_spin_lock+0x21/0x30 [10072.065537] process_one_work+0x2c8/0x700 [10072.065544] worker_thread+0x296/0x6e0 [10072.065549] ? rescuer_thread+0x570/0x570 [10072.065553] kthread+0x1d1/0x200 [10072.065559] ? set_kthread_struct+0x70/0x70 [10072.065564] ret_from_fork+0x1f/0x30 [10072.065402] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10072.071107] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10072.071187] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10072.071195] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10072.071198] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10072.071214] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10072.071218] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10072.071222] RAX: 0000000000004293 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10072.071224] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10072.071226] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10072.071228] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10072.071230] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10072.071233] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10072.071236] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10072.071238] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10072.071247] Call Trace: [10072.071263] [10072.071269] ? show_regs.cold.9+0x22/0x2f [10072.071278] ? watchdog_timer_fn.cold.6+0x62/0x10a [10072.071286] ? __hrtimer_run_queues+0x1c2/0x4d0 [10072.071290] ? watchdog+0x40/0x40 [10072.071293] ? hrtimer_interrupt+0x128/0x2c0 [10072.071297] ? smp_apic_timer_interrupt+0xa6/0x270 [10072.071307] ? apic_timer_interrupt+0xf/0x20 [10072.071311] [10072.071315] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10072.071319] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10072.071323] do_raw_spin_lock+0x177/0x190 [10072.071328] _raw_spin_lock+0x21/0x30 [10072.071333] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10072.081827] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10072.101821] ? do_raw_spin_unlock+0x75/0x190 [10072.107614] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10072.114009] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10072.119785] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10072.121158] ? do_wait_intr+0xf0/0xf0 [10072.121172] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10072.122078] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10072.123056] mdc_reint+0x2d/0x110 [mdc] [10072.123104] mdc_unlink+0x658/0x870 [mdc] [10072.123140] lmv_unlink+0x34d/0xbd0 [lmv] [10072.123183] ll_unlink+0x24d/0x7f0 [lustre] [10072.126829] ? woken_wake_function+0x30/0x30 [10072.128387] vfs_unlink+0x202/0x330 [10072.151588] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10072.153073] do_unlinkat+0x2fa/0x460 [10072.153084] __x64_sys_unlink+0x37/0x50 [10072.153088] do_syscall_64+0xc1/0x440 [10072.153100] entry_SYSCALL_64_after_hwframe+0x49/0xae [10072.153110] RIP: 0033:0x7f83e5f5c10b [10072.153153] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10072.153158] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10072.153164] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10072.153166] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10072.153169] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10072.153171] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10072.153173] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10072.693676] kthread+0x1d1/0x200 [10072.696332] ? set_kthread_struct+0x70/0x70 [10072.707280] ret_from_fork+0x1f/0x30 [10092.059729] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [10092.078438] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10092.122720] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10092.131711] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10092.138749] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10092.147329] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10092.166041] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10092.183810] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10092.190852] RDX: 0000000000005dce RSI: 0000000000000003 RDI: ffff9a61907912e0 [10092.202273] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10092.207754] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10092.213883] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10092.224641] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10092.234796] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10092.237368] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10092.252876] Call Trace: [10092.254185] [10092.256063] ? show_regs.cold.9+0x22/0x2f [10092.258717] ? watchdog_timer_fn.cold.6+0x62/0x10a [10092.264723] ? __hrtimer_run_queues+0x1c2/0x4d0 [10092.268408] ? watchdog+0x40/0x40 [10092.269746] ? hrtimer_interrupt+0x128/0x2c0 [10092.273005] ? smp_apic_timer_interrupt+0xa6/0x270 [10092.280793] ? apic_timer_interrupt+0xf/0x20 [10092.294591] [10092.296099] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10092.310714] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10092.314428] do_raw_spin_lock+0x177/0x190 [10092.316179] _raw_spin_lock+0x21/0x30 [10092.320820] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10092.328870] obd_mod_free+0x46/0x140 [ptlrpc] [10092.331560] mdc_commit_open+0x91/0xb0 [mdc] [10092.337395] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10092.345580] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10092.353214] after_reply+0xe2c/0x1780 [ptlrpc] [10092.358709] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10092.362212] ? __next_timer_interrupt+0x160/0x160 [10092.366780] ptlrpcd+0xa82/0xe40 [ptlrpc] [10092.369482] ? do_wait_intr+0xf0/0xf0 [10092.372825] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10092.378693] kthread+0x1d1/0x200 [10092.380785] ? set_kthread_struct+0x70/0x70 [10092.383897] ret_from_fork+0x1f/0x30 [10094.561049] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7581s! [10094.571464] Showing busy workqueues and worker pools: [10094.578906] workqueue events: flags=0x0 [10094.581750] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10094.599063] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10094.604176] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10094.608370] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10094.614247] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10094.625874] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10094.630422] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10094.633523] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10094.643459] workqueue events_long: flags=0x0 [10094.645578] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10094.652992] pending: lu_objects_flush [obdclass] [10094.658712] workqueue events_unbound: flags=0x2 [10094.660479] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10094.663529] pending: flush_memcg_stats_dwork, idle_cull_fn [10094.666806] workqueue events_power_efficient: flags=0x80 [10094.673570] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10094.699585] pending: neigh_periodic_work, do_cache_clean [10094.701705] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10094.706806] pending: neigh_periodic_work, check_lifetime [10094.713545] workqueue mm_percpu_wq: flags=0x8 [10094.720422] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10094.727607] pending: vmstat_update, lru_add_drain_per_cpu [10094.732296] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10094.746816] pending: vmstat_update [10094.749338] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10094.752237] pending: lru_add_drain_per_cpu BAR(48) [10094.761769] workqueue ipv6_addrconf: flags=0x40008 [10094.769164] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10094.775779] pending: addrconf_verify_work [10094.778486] workqueue knbd0-recv: flags=0x1a [10094.780695] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10094.795395] in-flight: 650:recv_work [10094.798204] workqueue ptlrpc_pinger: flags=0x2 [10094.803110] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10094.809681] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10094.818798] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7581s workers=4 idle: 36 466 29757 [10094.822907] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10084s workers=2 idle: 652 [10100.059041] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [10100.062047] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [10100.062060] Modules linked in: lustre(O) [10100.062999] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [10100.063007] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10100.063076] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10100.063081] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10100.063087] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10100.063377] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10100.063388] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10100.063392] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10100.063396] RAX: 000000000000517e RBX: ffff9a61907912e0 RCX: 0000000000000001 [10100.063398] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10100.063401] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10100.063407] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10100.063410] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10100.063413] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10100.063416] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10100.063418] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10100.063425] Call Trace: [10100.063431] [10100.063436] ? show_regs.cold.9+0x22/0x2f [10100.063446] ? watchdog_timer_fn.cold.6+0x62/0x10a [10100.063452] ? __hrtimer_run_queues+0x1c2/0x4d0 [10100.063455] ? watchdog+0x40/0x40 [10100.063460] ? hrtimer_interrupt+0x128/0x2c0 [10100.063464] ? smp_apic_timer_interrupt+0xa6/0x270 [10100.063471] ? apic_timer_interrupt+0xf/0x20 [10100.063475] [10100.063478] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10100.063482] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10100.063488] do_raw_spin_lock+0x177/0x190 [10100.063493] _raw_spin_lock+0x21/0x30 [10100.063497] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10100.064875] Modules linked in: lustre(O) osp(O) [10100.065443] process_one_work+0x2c8/0x700 [10100.065452] worker_thread+0x296/0x6e0 [10100.065458] ? rescuer_thread+0x570/0x570 [10100.065462] kthread+0x1d1/0x200 [10100.065466] ? set_kthread_struct+0x70/0x70 [10100.065470] ret_from_fork+0x1f/0x30 [10100.068086] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) [10100.071515] ofd(O) [10100.109319] ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10100.109710] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10100.109718] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10100.109721] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10100.109735] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10100.109740] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10100.109745] RAX: 000000000000122f RBX: ffff9a61907912e0 RCX: 0000000000000001 [10100.109748] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10100.109750] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10100.109753] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10100.109755] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10100.109760] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10100.109763] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10100.109765] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10100.109774] Call Trace: [10100.109780] [10100.109786] ? show_regs.cold.9+0x22/0x2f [10100.109795] ? watchdog_timer_fn.cold.6+0x62/0x10a [10100.109802] ? __hrtimer_run_queues+0x1c2/0x4d0 [10100.109807] ? watchdog+0x40/0x40 [10100.109811] ? hrtimer_interrupt+0x128/0x2c0 [10100.109814] ? smp_apic_timer_interrupt+0xa6/0x270 [10100.109821] ? apic_timer_interrupt+0xf/0x20 [10100.109839] [10100.109841] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10100.109846] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10100.109850] do_raw_spin_lock+0x177/0x190 [10100.109855] _raw_spin_lock+0x21/0x30 [10100.109860] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10100.110158] ? do_raw_spin_unlock+0x75/0x190 [10100.110164] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10100.110766] ? do_wait_intr+0xf0/0xf0 [10100.110775] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10100.111068] mdc_reint+0x2d/0x110 [mdc] [10100.111108] mdc_unlink+0x658/0x870 [mdc] [10100.111133] lmv_unlink+0x34d/0xbd0 [lmv] [10100.111172] ll_unlink+0x24d/0x7f0 [lustre] [10100.111753] vfs_unlink+0x202/0x330 [10100.111761] do_unlinkat+0x2fa/0x460 [10100.111768] __x64_sys_unlink+0x37/0x50 [10100.111771] do_syscall_64+0xc1/0x440 [10100.111777] entry_SYSCALL_64_after_hwframe+0x49/0xae [10100.111785] RIP: 0033:0x7f83e5f5c10b [10100.111792] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10100.111797] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10100.111802] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10100.111804] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10100.111806] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10100.111807] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10100.111809] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10100.619212] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10100.650568] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10100.657522] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10100.672394] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10100.678313] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10100.691139] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10100.695090] RAX: 000000000000313f RBX: ffff9a61907912e0 RCX: 0000000000000001 [10100.699799] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10100.706698] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10100.713485] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10100.715707] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10100.721767] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10100.727519] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10100.730536] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10100.732840] Call Trace: [10100.733510] [10100.735921] ? show_regs.cold.9+0x22/0x2f [10100.741155] ? watchdog_timer_fn.cold.6+0x62/0x10a [10100.745523] ? __hrtimer_run_queues+0x1c2/0x4d0 [10100.748543] ? watchdog+0x40/0x40 [10100.753901] ? hrtimer_interrupt+0x128/0x2c0 [10100.757713] ? smp_apic_timer_interrupt+0xa6/0x270 [10100.765715] ? apic_timer_interrupt+0xf/0x20 [10100.767817] [10100.770345] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10100.779342] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10100.783321] do_raw_spin_lock+0x177/0x190 [10100.788006] _raw_spin_lock+0x21/0x30 [10100.791024] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10100.794761] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10100.801562] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10100.807051] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10100.817734] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10100.820278] ? woken_wake_function+0x30/0x30 [10100.827271] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10100.833204] kthread+0x1d1/0x200 [10100.834381] ? set_kthread_struct+0x70/0x70 [10100.838636] ret_from_fork+0x1f/0x30 [10120.056039] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [10120.059928] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10120.078771] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10120.083043] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10120.087226] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10120.089616] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10120.097432] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10120.100704] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10120.103793] RDX: 00000000000068e5 RSI: 0000000000000003 RDI: ffff9a61907912e0 [10120.106846] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10120.110205] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10120.113230] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10120.116463] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10120.120326] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10120.122906] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10120.125718] Call Trace: [10120.126566] [10120.127480] ? show_regs.cold.9+0x22/0x2f [10120.129350] ? watchdog_timer_fn.cold.6+0x62/0x10a [10120.131326] ? __hrtimer_run_queues+0x1c2/0x4d0 [10120.132878] ? watchdog+0x40/0x40 [10120.133761] ? hrtimer_interrupt+0x128/0x2c0 [10120.135456] ? smp_apic_timer_interrupt+0xa6/0x270 [10120.137385] ? apic_timer_interrupt+0xf/0x20 [10120.139347] [10120.140292] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10120.142542] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10120.144610] do_raw_spin_lock+0x177/0x190 [10120.146432] _raw_spin_lock+0x21/0x30 [10120.147824] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10120.150195] obd_mod_free+0x46/0x140 [ptlrpc] [10120.152663] mdc_commit_open+0x91/0xb0 [mdc] [10120.154656] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10120.156672] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10120.159538] after_reply+0xe2c/0x1780 [ptlrpc] [10120.161544] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10120.163780] ? __next_timer_interrupt+0x160/0x160 [10120.165854] ptlrpcd+0xa82/0xe40 [ptlrpc] [10120.167940] ? do_wait_intr+0xf0/0xf0 [10120.169525] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10120.171932] kthread+0x1d1/0x200 [10120.173380] ? set_kthread_struct+0x70/0x70 [10120.175257] ret_from_fork+0x1f/0x30 [10125.281220] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7612s! [10125.286046] Showing busy workqueues and worker pools: [10125.289350] workqueue events: flags=0x0 [10125.290566] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10125.294177] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10125.298802] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10125.302101] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10125.304897] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10125.308237] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10125.311284] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10125.314118] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10125.318009] workqueue events_long: flags=0x0 [10125.319593] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10125.322451] pending: lu_objects_flush [obdclass] [10125.324478] workqueue events_unbound: flags=0x2 [10125.325870] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10125.328581] pending: flush_memcg_stats_dwork, idle_cull_fn [10125.330902] workqueue events_power_efficient: flags=0x80 [10125.333388] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10125.336135] pending: neigh_periodic_work, do_cache_clean [10125.338521] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10125.341175] pending: neigh_periodic_work, check_lifetime [10125.343048] workqueue mm_percpu_wq: flags=0x8 [10125.344246] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10125.347371] pending: vmstat_update, lru_add_drain_per_cpu [10125.350189] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10125.353795] pending: vmstat_update [10125.355257] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10125.358071] pending: lru_add_drain_per_cpu BAR(48) [10125.359545] workqueue ipv6_addrconf: flags=0x40008 [10125.361325] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10125.363893] pending: addrconf_verify_work [10125.365708] workqueue knbd0-recv: flags=0x1a [10125.367249] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10125.369875] in-flight: 650:recv_work [10125.371359] workqueue ptlrpc_pinger: flags=0x2 [10125.373199] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10125.375693] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10125.378288] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7612s workers=4 idle: 36 466 29757 [10125.381467] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10115s workers=2 idle: 652 [10128.060126] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [10128.062760] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [10128.066151] Modules linked in: [10128.066749] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [10128.066756] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10128.067581] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10128.067588] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10128.067591] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10128.067609] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10128.067614] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10128.067619] RAX: 0000000000007a92 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10128.067622] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10128.067624] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10128.067626] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10128.067628] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10128.067633] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10128.067635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10128.067638] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10128.067647] Call Trace: [10128.067654] [10128.067662] ? show_regs.cold.9+0x22/0x2f [10128.067672] ? watchdog_timer_fn.cold.6+0x62/0x10a [10128.067679] ? __hrtimer_run_queues+0x1c2/0x4d0 [10128.067683] ? watchdog+0x40/0x40 [10128.067688] ? hrtimer_interrupt+0x128/0x2c0 [10128.067692] ? smp_apic_timer_interrupt+0xa6/0x270 [10128.067700] ? apic_timer_interrupt+0xf/0x20 [10128.067714] [10128.067718] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10128.067724] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10128.067730] do_raw_spin_lock+0x177/0x190 [10128.067735] _raw_spin_lock+0x21/0x30 [10128.067739] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10128.069676] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10128.070006] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10128.070248] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10128.070498] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10128.070737] Modules linked in: lustre(O) osp(O) ofd(O) [10128.070740] ? woken_wake_function+0x30/0x30 [10128.070750] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10128.070813] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10128.070820] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10128.070823] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10128.070834] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10128.070839] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10128.070844] RAX: 000000000000390d RBX: ffff9a61907912e0 RCX: 0000000000000001 [10128.070846] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10128.070849] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10128.070851] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10128.070853] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10128.070857] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10128.070757] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10128.070860] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10128.070866] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10128.070875] Call Trace: [10128.070880] [10128.070884] ? show_regs.cold.9+0x22/0x2f [10128.070895] ? watchdog_timer_fn.cold.6+0x62/0x10a [10128.070902] ? __hrtimer_run_queues+0x1c2/0x4d0 [10128.070907] ? watchdog+0x40/0x40 [10128.070911] ? hrtimer_interrupt+0x128/0x2c0 [10128.070915] ? smp_apic_timer_interrupt+0xa6/0x270 [10128.070924] ? apic_timer_interrupt+0xf/0x20 [10128.070929] [10128.070932] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10128.070936] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10128.070940] do_raw_spin_lock+0x177/0x190 [10128.070945] _raw_spin_lock+0x21/0x30 [10128.070992] kthread+0x1d1/0x200 [10128.070998] ? set_kthread_struct+0x70/0x70 [10128.071002] ret_from_fork+0x1f/0x30 [10128.070951] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10128.073683] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10128.073930] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10128.073938] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10128.073945] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10128.074238] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10128.074594] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10128.074597] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10128.074602] RAX: 000000000000160e RBX: ffff9a61907912e0 RCX: 0000000000000001 [10128.074604] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10128.074606] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10128.074609] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10128.074611] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10128.074614] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10128.074617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10128.074619] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10128.074627] Call Trace: [10128.074633] [10128.074638] ? show_regs.cold.9+0x22/0x2f [10128.074649] ? watchdog_timer_fn.cold.6+0x62/0x10a [10128.074655] ? __hrtimer_run_queues+0x1c2/0x4d0 [10128.074659] ? watchdog+0x40/0x40 [10128.074669] ? hrtimer_interrupt+0x128/0x2c0 [10128.074673] ? smp_apic_timer_interrupt+0xa6/0x270 [10128.074681] ? apic_timer_interrupt+0xf/0x20 [10128.074686] [10128.074689] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10128.074694] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10128.074700] do_raw_spin_lock+0x177/0x190 [10128.074714] _raw_spin_lock+0x21/0x30 [10128.074722] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10128.075051] process_one_work+0x2c8/0x700 [10128.075059] worker_thread+0x296/0x6e0 [10128.075064] ? rescuer_thread+0x570/0x570 [10128.075068] kthread+0x1d1/0x200 [10128.075073] ? set_kthread_struct+0x70/0x70 [10128.075076] ret_from_fork+0x1f/0x30 [10128.552395] ? do_raw_spin_unlock+0x75/0x190 [10128.554858] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10128.556991] ? do_wait_intr+0xf0/0xf0 [10128.559357] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10128.560919] mdc_reint+0x2d/0x110 [mdc] [10128.562131] mdc_unlink+0x658/0x870 [mdc] [10128.563220] lmv_unlink+0x34d/0xbd0 [lmv] [10128.564219] ll_unlink+0x24d/0x7f0 [lustre] [10128.565532] vfs_unlink+0x202/0x330 [10128.569893] do_unlinkat+0x2fa/0x460 [10128.571842] __x64_sys_unlink+0x37/0x50 [10128.573331] do_syscall_64+0xc1/0x440 [10128.574477] entry_SYSCALL_64_after_hwframe+0x49/0xae [10128.581839] RIP: 0033:0x7f83e5f5c10b [10128.583259] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10128.596582] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10128.602118] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10128.605498] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10128.611476] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10128.614649] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10128.620070] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10141.161277] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [10141.165299] rcu: 0-....: (5111172 ticks this GP) idle=30a/1/0x4000000000000000 softirq=128519/128519 fqs=126395 [10141.179117] (detected by 1, t=8160202 jiffies, g=262441, q=59724) [10141.183920] Sending NMI from CPU 1 to CPUs 0: [10141.195723] NMI backtrace for cpu 0 [10141.195731] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10141.195733] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10141.195735] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10141.195736] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10141.195739] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 [10141.195741] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10141.195743] RDX: 00000000000067f1 RSI: 0000000000000003 RDI: ffff9a61907912e0 [10141.195744] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10141.195746] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10141.195748] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10141.195749] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10141.195750] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10141.195751] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10141.195753] Call Trace: [10141.195754] [10141.195755] ? show_regs.cold.9+0x22/0x2f [10141.195756] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10141.195757] ? nmi_cpu_backtrace_handler+0x15/0x40 [10141.195758] ? nmi_handle+0xbd/0x210 [10141.195759] ? default_do_nmi+0x4b/0x1b0 [10141.195760] ? do_nmi+0x280/0x380 [10141.195761] ? end_repeat_nmi+0x16/0x1f [10141.195762] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10141.195763] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10141.195765] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10141.195766] [10141.195767] do_raw_spin_lock+0x177/0x190 [10141.195768] _raw_spin_lock+0x21/0x30 [10141.195769] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10141.195770] obd_mod_free+0x46/0x140 [ptlrpc] [10141.195771] mdc_commit_open+0x91/0xb0 [mdc] [10141.195772] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10141.195773] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10141.195774] after_reply+0xe2c/0x1780 [ptlrpc] [10141.195775] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10141.195776] ? __next_timer_interrupt+0x160/0x160 [10141.195777] ptlrpcd+0xa82/0xe40 [ptlrpc] [10141.195778] ? do_wait_intr+0xf0/0xf0 [10141.195779] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10141.195780] kthread+0x1d1/0x200 [10141.195781] ? set_kthread_struct+0x70/0x70 [10141.195782] ret_from_fork+0x1f/0x30 [10141.201908] rcu: rcu_sched kthread starved for 7629389 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [10141.357995] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10141.362089] rcu: RCU grace-period kthread stack dump: [10141.364592] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10141.368606] Call Trace: [10141.369724] __schedule+0x351/0xcb0 [10141.371390] schedule+0xc0/0x180 [10141.372962] schedule_timeout+0xb4/0x190 [10141.374897] ? do_raw_spin_unlock+0x75/0x190 [10141.380275] ? __next_timer_interrupt+0x160/0x160 [10141.385812] ? prepare_to_swait_event+0xf3/0x140 [10141.393569] rcu_gp_kthread+0x8dd/0x15b0 [10141.395599] ? force_qs_rnp+0x2c0/0x2c0 [10141.398245] kthread+0x1d1/0x200 [10141.403127] ? set_kthread_struct+0x70/0x70 [10141.408376] ret_from_fork+0x1f/0x30 [10141.412297] rcu: Stack dump where RCU GP kthread last ran: [10141.416230] NMI backtrace for cpu 1 [10141.419871] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10141.428246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10141.434612] Call Trace: [10141.435502] [10141.439060] ? dump_stack+0xbb/0x10e [10141.441329] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10141.447366] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10141.450617] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10141.453611] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10141.458105] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10141.464048] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [10141.466942] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [10141.470301] ? update_process_times+0x6d/0xc0 [10141.476889] ? tick_sched_handle.isra.18+0x27/0x90 [10141.482697] ? tick_sched_timer+0x53/0xb0 [10141.485179] ? __hrtimer_run_queues+0x1c2/0x4d0 [10141.491884] ? tick_sched_handle.isra.18+0x90/0x90 [10141.494616] ? hrtimer_interrupt+0x128/0x2c0 [10141.501227] ? smp_apic_timer_interrupt+0xa6/0x270 [10141.503505] ? apic_timer_interrupt+0xf/0x20 [10141.506720] [10141.507791] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10141.510316] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10141.512860] ? do_raw_spin_lock+0x177/0x190 [10141.514815] ? _raw_spin_lock+0x21/0x30 [10141.516437] ? ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10141.519391] ? ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10141.522435] ? ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10141.525117] ? ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10141.528691] ? ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10141.531488] ? woken_wake_function+0x30/0x30 [10141.533372] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10141.536346] ? kthread+0x1d1/0x200 [10141.538043] ? set_kthread_struct+0x70/0x70 [10141.539964] ? ret_from_fork+0x1f/0x30 [10155.489042] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7642s! [10155.492801] Showing busy workqueues and worker pools: [10155.494990] workqueue events: flags=0x0 [10155.497381] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10155.501043] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10155.505287] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10155.507939] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10155.510726] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10155.513893] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10155.517725] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10155.520735] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10155.524800] workqueue events_long: flags=0x0 [10155.526405] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10155.529604] pending: lu_objects_flush [obdclass] [10155.532097] workqueue events_unbound: flags=0x2 [10155.534150] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10155.536632] pending: flush_memcg_stats_dwork, idle_cull_fn [10155.538814] workqueue events_power_efficient: flags=0x80 [10155.540714] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10155.543350] pending: neigh_periodic_work, do_cache_clean [10155.545911] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10155.548956] pending: neigh_periodic_work, check_lifetime [10155.551377] workqueue mm_percpu_wq: flags=0x8 [10155.553474] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10155.556244] pending: vmstat_update, lru_add_drain_per_cpu [10155.558961] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10155.561784] pending: vmstat_update [10155.563793] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10155.566630] pending: lru_add_drain_per_cpu BAR(48) [10155.568912] workqueue ipv6_addrconf: flags=0x40008 [10155.571015] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10155.573468] pending: addrconf_verify_work [10155.575483] workqueue knbd0-recv: flags=0x1a [10155.577420] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10155.580822] in-flight: 650:recv_work [10155.583048] workqueue ptlrpc_pinger: flags=0x2 [10155.585638] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10155.588898] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10155.591544] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7642s workers=4 idle: 36 466 29757 [10155.595017] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10145s workers=2 idle: 652 [10156.062191] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [10156.065665] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [10156.067454] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10156.067533] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10156.067541] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10156.067544] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [10156.067558] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [10156.067562] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [10156.067567] RAX: 0000000000001f74 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10156.067569] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10156.067571] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10156.067573] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10156.067575] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10156.067578] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10156.067581] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10156.067583] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10156.067591] Call Trace: [10156.067597] [10156.067602] ? show_regs.cold.9+0x22/0x2f [10156.067611] ? watchdog_timer_fn.cold.6+0x62/0x10a [10156.067617] ? __hrtimer_run_queues+0x1c2/0x4d0 [10156.067622] ? watchdog+0x40/0x40 [10156.067626] ? hrtimer_interrupt+0x128/0x2c0 [10156.067630] ? smp_apic_timer_interrupt+0xa6/0x270 [10156.067705] ? apic_timer_interrupt+0xf/0x20 [10156.067710] [10156.067714] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [10156.067719] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10156.067723] do_raw_spin_lock+0x177/0x190 [10156.067729] _raw_spin_lock+0x21/0x30 [10156.067735] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10156.067998] ? do_raw_spin_unlock+0x75/0x190 [10156.068004] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10156.068225] ? do_wait_intr+0xf0/0xf0 [10156.068233] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10156.068429] mdc_reint+0x2d/0x110 [mdc] [10156.072103] mdc_unlink+0x658/0x870 [mdc] [10156.072130] lmv_unlink+0x34d/0xbd0 [lmv] [10156.072165] ll_unlink+0x24d/0x7f0 [lustre] [10156.072273] vfs_unlink+0x202/0x330 [10156.072282] do_unlinkat+0x2fa/0x460 [10156.072288] __x64_sys_unlink+0x37/0x50 [10156.072292] do_syscall_64+0xc1/0x440 [10156.072298] entry_SYSCALL_64_after_hwframe+0x49/0xae [10156.072306] RIP: 0033:0x7f83e5f5c10b [10156.072314] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10156.072317] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10156.072323] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10156.072325] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10156.072327] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10156.072330] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10156.072332] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10156.366781] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10156.390510] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10156.401241] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10156.405864] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10156.410578] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10156.418566] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10156.430991] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10156.436345] RAX: 0000000000007e87 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10156.440134] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10156.443200] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10156.446177] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10156.449097] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10156.451866] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10156.456524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10156.459520] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10156.463272] Call Trace: [10156.464855] [10156.465776] ? show_regs.cold.9+0x22/0x2f [10156.467982] ? watchdog_timer_fn.cold.6+0x62/0x10a [10156.470578] ? __hrtimer_run_queues+0x1c2/0x4d0 [10156.473274] ? watchdog+0x40/0x40 [10156.474743] ? hrtimer_interrupt+0x128/0x2c0 [10156.476546] ? smp_apic_timer_interrupt+0xa6/0x270 [10156.479000] ? apic_timer_interrupt+0xf/0x20 [10156.481628] [10156.483512] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10156.486967] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10156.488723] do_raw_spin_lock+0x177/0x190 [10156.489950] _raw_spin_lock+0x21/0x30 [10156.494456] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10156.498506] process_one_work+0x2c8/0x700 [10156.502326] worker_thread+0x296/0x6e0 [10156.504344] ? rescuer_thread+0x570/0x570 [10156.506674] kthread+0x1d1/0x200 [10156.508085] ? set_kthread_struct+0x70/0x70 [10156.510754] ret_from_fork+0x1f/0x30 [10168.060729] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [10168.063696] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10168.090132] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10168.097195] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10168.108752] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10168.113076] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10168.123315] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10168.138155] RAX: 0000000000007885 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10168.145954] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10168.151433] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10168.155184] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10168.159670] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10168.169673] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10168.175618] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10168.178634] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10168.183284] Call Trace: [10168.186395] [10168.192139] ? show_regs.cold.9+0x22/0x2f [10168.196231] ? watchdog_timer_fn.cold.6+0x62/0x10a [10168.205004] ? __hrtimer_run_queues+0x1c2/0x4d0 [10168.210007] ? watchdog+0x40/0x40 [10168.213211] ? hrtimer_interrupt+0x128/0x2c0 [10168.217438] ? smp_apic_timer_interrupt+0xa6/0x270 [10168.224468] ? apic_timer_interrupt+0xf/0x20 [10168.229631] [10168.230442] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10168.239504] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10168.243411] do_raw_spin_lock+0x177/0x190 [10168.245982] _raw_spin_lock+0x21/0x30 [10168.248213] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10168.252126] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10168.254807] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10168.257278] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10168.259777] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10168.262596] ? woken_wake_function+0x30/0x30 [10168.264436] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10168.267243] kthread+0x1d1/0x200 [10168.269084] ? set_kthread_struct+0x70/0x70 [10168.270971] ret_from_fork+0x1f/0x30 [10180.058321] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [10180.062992] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10180.095597] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10180.100752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10180.106571] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10180.113019] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10180.130243] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10180.137816] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10180.146029] RDX: 00000000000070ae RSI: 0000000000000003 RDI: ffff9a61907912e0 [10180.148450] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10180.150818] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10180.153467] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10180.155863] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10180.158978] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10180.161768] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10180.170728] Call Trace: [10180.171981] [10180.173268] ? show_regs.cold.9+0x22/0x2f [10180.182020] ? watchdog_timer_fn.cold.6+0x62/0x10a [10180.183196] ? __hrtimer_run_queues+0x1c2/0x4d0 [10180.187540] ? watchdog+0x40/0x40 [10180.189702] ? hrtimer_interrupt+0x128/0x2c0 [10180.192103] ? smp_apic_timer_interrupt+0xa6/0x270 [10180.193796] ? apic_timer_interrupt+0xf/0x20 [10180.195881] [10180.199029] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10180.203368] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10180.206575] do_raw_spin_lock+0x177/0x190 [10180.214596] _raw_spin_lock+0x21/0x30 [10180.216128] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10180.217428] obd_mod_free+0x46/0x140 [ptlrpc] [10180.218934] mdc_commit_open+0x91/0xb0 [mdc] [10180.220656] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10180.222900] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10180.225262] after_reply+0xe2c/0x1780 [ptlrpc] [10180.227281] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10180.230944] ? __next_timer_interrupt+0x160/0x160 [10180.232864] ptlrpcd+0xa82/0xe40 [ptlrpc] [10180.234963] ? do_wait_intr+0xf0/0xf0 [10180.238231] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10180.242482] kthread+0x1d1/0x200 [10180.245722] ? set_kthread_struct+0x70/0x70 [10180.248788] ret_from_fork+0x1f/0x30 [10184.062214] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [10184.062280] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [10184.068016] Modules linked in: [10184.089497] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10184.089579] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10184.089586] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10184.089592] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10184.092914] lustre(O) [10184.124074] [10184.124083] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10184.124102] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10184.124106] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10184.124111] RAX: 000000000000571d RBX: ffff9a61907912e0 RCX: 0000000000000001 [10184.124113] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10184.124115] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10184.124117] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10184.124119] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10184.124122] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10184.124125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10184.124127] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10184.124205] Call Trace: [10184.131143] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul [10184.136119] [10184.136205] ? show_regs.cold.9+0x22/0x2f [10184.139269] crc32_pclmul [10184.140793] ? watchdog_timer_fn.cold.6+0x62/0x10a [10184.140809] ? __hrtimer_run_queues+0x1c2/0x4d0 [10184.140814] ? watchdog+0x40/0x40 [10184.140819] ? hrtimer_interrupt+0x128/0x2c0 [10184.140825] ? smp_apic_timer_interrupt+0xa6/0x270 [10184.141778] crc32c_intel [10184.144802] ? apic_timer_interrupt+0xf/0x20 [10184.155901] ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10184.155924] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10184.155932] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10184.155935] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10184.155948] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10184.155952] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10184.155955] RAX: 0000000000003e84 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10184.155958] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10184.155960] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10184.155963] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10184.155965] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10184.155969] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10184.155972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10184.155974] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10184.155989] Call Trace: [10184.155994] [10184.156000] ? show_regs.cold.9+0x22/0x2f [10184.156023] ? watchdog_timer_fn.cold.6+0x62/0x10a [10184.156036] ? __hrtimer_run_queues+0x1c2/0x4d0 [10184.156040] ? watchdog+0x40/0x40 [10184.156045] ? hrtimer_interrupt+0x128/0x2c0 [10184.156048] ? smp_apic_timer_interrupt+0xa6/0x270 [10184.156055] ? apic_timer_interrupt+0xf/0x20 [10184.156060] [10184.156062] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10184.156067] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10184.156070] do_raw_spin_lock+0x177/0x190 [10184.156074] _raw_spin_lock+0x21/0x30 [10184.156081] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10184.156559] ? do_raw_spin_unlock+0x75/0x190 [10184.156566] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10184.156773] ? do_wait_intr+0xf0/0xf0 [10184.156780] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10184.156977] mdc_reint+0x2d/0x110 [mdc] [10184.157203] mdc_unlink+0x658/0x870 [mdc] [10184.157232] lmv_unlink+0x34d/0xbd0 [lmv] [10184.157261] ll_unlink+0x24d/0x7f0 [lustre] [10184.157390] vfs_unlink+0x202/0x330 [10184.157400] do_unlinkat+0x2fa/0x460 [10184.157405] __x64_sys_unlink+0x37/0x50 [10184.157409] do_syscall_64+0xc1/0x440 [10184.157415] entry_SYSCALL_64_after_hwframe+0x49/0xae [10184.157422] RIP: 0033:0x7f83e5f5c10b [10184.157429] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10184.157433] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10184.157438] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10184.157440] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10184.157443] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10184.157445] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10184.157447] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10184.566827] [10184.568003] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10184.573992] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10184.583756] do_raw_spin_lock+0x177/0x190 [10184.586993] _raw_spin_lock+0x21/0x30 [10184.589012] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10184.595472] process_one_work+0x2c8/0x700 [10184.597629] worker_thread+0x296/0x6e0 [10184.599543] ? rescuer_thread+0x570/0x570 [10184.603771] kthread+0x1d1/0x200 [10184.606054] ? set_kthread_struct+0x70/0x70 [10184.608050] ret_from_fork+0x1f/0x30 [10185.696093] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7672s! [10185.700648] Showing busy workqueues and worker pools: [10185.706169] workqueue events: flags=0x0 [10185.709563] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10185.713316] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10185.727472] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10185.734586] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10185.738875] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10185.746539] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10185.752249] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10185.756257] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10185.765379] workqueue events_long: flags=0x0 [10185.767587] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10185.773214] pending: lu_objects_flush [obdclass] [10185.777596] workqueue events_unbound: flags=0x2 [10185.781237] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10185.785498] pending: flush_memcg_stats_dwork, idle_cull_fn [10185.793465] workqueue events_power_efficient: flags=0x80 [10185.800180] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10185.808942] pending: neigh_periodic_work, do_cache_clean [10185.812931] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10185.820036] pending: neigh_periodic_work, check_lifetime [10185.825803] workqueue mm_percpu_wq: flags=0x8 [10185.829197] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10185.834643] pending: vmstat_update, lru_add_drain_per_cpu [10185.837491] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10185.839910] pending: vmstat_update [10185.841803] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10185.845336] pending: lru_add_drain_per_cpu BAR(48) [10185.851912] workqueue ipv6_addrconf: flags=0x40008 [10185.859013] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10185.864539] pending: addrconf_verify_work [10185.866505] workqueue knbd0-recv: flags=0x1a [10185.870340] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10185.874403] in-flight: 650:recv_work [10185.876504] workqueue ptlrpc_pinger: flags=0x2 [10185.879404] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10185.882956] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10185.886313] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7672s workers=4 idle: 36 466 29757 [10185.890448] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10175s workers=2 idle: 652 [10196.061284] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [10196.073607] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10196.103022] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10196.107450] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10196.111223] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10196.113430] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10196.120352] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10196.122890] RAX: 000000000000340a RBX: ffff9a61907912e0 RCX: 0000000000000001 [10196.125939] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10196.134020] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10196.140326] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10196.143226] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10196.151888] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10196.159194] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10196.172426] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10196.183415] Call Trace: [10196.184351] [10196.186960] ? show_regs.cold.9+0x22/0x2f [10196.191875] ? watchdog_timer_fn.cold.6+0x62/0x10a [10196.199110] ? __hrtimer_run_queues+0x1c2/0x4d0 [10196.203904] ? watchdog+0x40/0x40 [10196.206298] ? hrtimer_interrupt+0x128/0x2c0 [10196.211223] ? smp_apic_timer_interrupt+0xa6/0x270 [10196.214076] ? apic_timer_interrupt+0xf/0x20 [10196.220182] [10196.221150] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10196.227040] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10196.235474] do_raw_spin_lock+0x177/0x190 [10196.237734] _raw_spin_lock+0x21/0x30 [10196.242697] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10196.250224] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10196.257678] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10196.264682] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10196.274100] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10196.280454] ? woken_wake_function+0x30/0x30 [10196.282605] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10196.295853] kthread+0x1d1/0x200 [10196.297909] ? set_kthread_struct+0x70/0x70 [10196.304083] ret_from_fork+0x1f/0x30 [10212.057155] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [10212.060235] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [10212.061055] Modules linked in: lustre(O) [10212.062066] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [10212.062075] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10212.062152] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10212.062157] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10212.062160] RIP: 0010:native_safe_halt+0x2e/0x50 [10212.062174] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10212.062179] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10212.062183] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10212.062186] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10212.062188] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10212.062191] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10212.062193] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10212.062197] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10212.062200] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10212.062202] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10212.062211] Call Trace: [10212.062216] [10212.062221] ? show_regs.cold.9+0x22/0x2f [10212.062232] ? watchdog_timer_fn.cold.6+0x62/0x10a [10212.062238] ? __hrtimer_run_queues+0x1c2/0x4d0 [10212.062243] ? watchdog+0x40/0x40 [10212.062247] ? hrtimer_interrupt+0x128/0x2c0 [10212.062250] ? smp_apic_timer_interrupt+0xa6/0x270 [10212.062257] ? apic_timer_interrupt+0xf/0x20 [10212.062262] [10212.062265] ? native_safe_halt+0x2e/0x50 [10212.062269] kvm_wait+0xcd/0xe0 [10212.062277] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10212.062284] do_raw_spin_lock+0x177/0x190 [10212.062289] _raw_spin_lock+0x21/0x30 [10212.062294] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10212.062563] ? do_raw_spin_unlock+0x75/0x190 [10212.062570] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10212.062764] ? do_wait_intr+0xf0/0xf0 [10212.062789] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10212.062998] mdc_reint+0x2d/0x110 [mdc] [10212.063105] mdc_unlink+0x658/0x870 [mdc] [10212.063133] lmv_unlink+0x34d/0xbd0 [lmv] [10212.063166] ll_unlink+0x24d/0x7f0 [lustre] [10212.063288] vfs_unlink+0x202/0x330 [10212.063297] do_unlinkat+0x2fa/0x460 [10212.063302] __x64_sys_unlink+0x37/0x50 [10212.063305] do_syscall_64+0xc1/0x440 [10212.063312] entry_SYSCALL_64_after_hwframe+0x49/0xae [10212.063319] RIP: 0033:0x7f83e5f5c10b [10212.063326] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10212.063330] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10212.063334] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10212.063337] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10212.063339] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10212.063341] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10212.063345] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10212.071774] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10212.071859] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10212.071866] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10212.071877] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10212.072173] RIP: 0010:native_safe_halt+0x2e/0x50 [10212.072184] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10212.072189] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10212.072193] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10212.072196] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10212.072198] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10212.072200] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10212.072202] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10212.072206] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10212.072209] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10212.072211] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10212.072220] Call Trace: [10212.072226] [10212.072231] ? show_regs.cold.9+0x22/0x2f [10212.072241] ? watchdog_timer_fn.cold.6+0x62/0x10a [10212.072248] ? __hrtimer_run_queues+0x1c2/0x4d0 [10212.072253] ? watchdog+0x40/0x40 [10212.072256] ? hrtimer_interrupt+0x128/0x2c0 [10212.072260] ? smp_apic_timer_interrupt+0xa6/0x270 [10212.072268] ? apic_timer_interrupt+0xf/0x20 [10212.072272] [10212.072275] ? native_safe_halt+0x2e/0x50 [10212.072285] kvm_wait+0xcd/0xe0 [10212.072294] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10212.072302] do_raw_spin_lock+0x177/0x190 [10212.072306] _raw_spin_lock+0x21/0x30 [10212.072310] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10212.072530] process_one_work+0x2c8/0x700 [10212.072539] worker_thread+0x296/0x6e0 [10212.072548] ? rescuer_thread+0x570/0x570 [10212.072552] kthread+0x1d1/0x200 [10212.072556] ? set_kthread_struct+0x70/0x70 [10212.072560] ret_from_fork+0x1f/0x30 [10212.717655] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10212.767575] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10212.780560] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10212.788373] RIP: 0010:native_safe_halt+0x2e/0x50 [10212.797377] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10212.820653] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10212.826424] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10212.835382] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [10212.846505] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10212.852601] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10212.861773] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10212.865296] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10212.876387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10212.884353] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10212.890325] Call Trace: [10212.891609] [10212.894495] ? show_regs.cold.9+0x22/0x2f [10212.897208] ? watchdog_timer_fn.cold.6+0x62/0x10a [10212.903722] ? __hrtimer_run_queues+0x1c2/0x4d0 [10212.905344] ? watchdog+0x40/0x40 [10212.911207] ? hrtimer_interrupt+0x128/0x2c0 [10212.913640] ? smp_apic_timer_interrupt+0xa6/0x270 [10212.920198] ? apic_timer_interrupt+0xf/0x20 [10212.921717] [10212.927673] ? native_safe_halt+0x2e/0x50 [10212.930463] kvm_wait+0xcd/0xe0 [10212.936411] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10212.945967] do_raw_spin_lock+0x177/0x190 [10212.947411] _raw_spin_lock+0x21/0x30 [10212.948807] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10212.964358] obd_mod_free+0x46/0x140 [ptlrpc] [10212.967721] mdc_commit_open+0x91/0xb0 [mdc] [10212.972547] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10212.980675] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10212.990493] after_reply+0xe2c/0x1780 [ptlrpc] [10212.995688] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10212.997869] ? __next_timer_interrupt+0x160/0x160 [10213.009579] ptlrpcd+0xa82/0xe40 [ptlrpc] [10213.016606] ? do_wait_intr+0xf0/0xf0 [10213.019642] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10213.022979] kthread+0x1d1/0x200 [10213.026482] ? set_kthread_struct+0x70/0x70 [10213.030571] ret_from_fork+0x1f/0x30 [10215.905117] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7702s! [10215.907737] Showing busy workqueues and worker pools: [10215.909886] workqueue events: flags=0x0 [10215.911605] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10215.914652] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10215.918605] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10215.921772] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10215.925022] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10215.928267] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10215.931777] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10215.934794] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10215.939151] workqueue events_long: flags=0x0 [10215.940848] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10215.943795] pending: lu_objects_flush [obdclass] [10215.946491] workqueue events_unbound: flags=0x2 [10215.949025] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10215.952228] pending: flush_memcg_stats_dwork, idle_cull_fn [10215.955552] workqueue events_power_efficient: flags=0x80 [10215.965345] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10215.972502] pending: neigh_periodic_work, do_cache_clean [10215.975761] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10215.981375] pending: neigh_periodic_work, check_lifetime [10215.987649] workqueue mm_percpu_wq: flags=0x8 [10215.992960] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10215.996691] pending: vmstat_update, lru_add_drain_per_cpu [10216.000256] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10216.003908] pending: vmstat_update [10216.007464] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10216.011360] pending: lru_add_drain_per_cpu BAR(48) [10216.014546] workqueue ipv6_addrconf: flags=0x40008 [10216.017188] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10216.020969] pending: addrconf_verify_work [10216.022721] workqueue knbd0-recv: flags=0x1a [10216.024538] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10216.027411] in-flight: 650:recv_work [10216.028659] workqueue ptlrpc_pinger: flags=0x2 [10216.033356] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10216.036161] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10216.039995] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7702s workers=4 idle: 36 466 29757 [10216.044617] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10206s workers=2 idle: 652 [10224.059367] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [10224.067593] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10224.125748] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10224.137112] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10224.143886] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10224.148655] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10224.166944] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [10224.179139] RAX: 00000000000055e4 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10224.187548] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10224.194520] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10224.202566] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10224.207167] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10224.214308] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10224.221192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10224.225739] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10224.233844] Call Trace: [10224.234912] [10224.237466] ? show_regs.cold.9+0x22/0x2f [10224.240261] ? watchdog_timer_fn.cold.6+0x62/0x10a [10224.242399] ? __hrtimer_run_queues+0x1c2/0x4d0 [10224.243788] ? watchdog+0x40/0x40 [10224.245536] ? hrtimer_interrupt+0x128/0x2c0 [10224.248813] ? smp_apic_timer_interrupt+0xa6/0x270 [10224.251543] ? apic_timer_interrupt+0xf/0x20 [10224.253418] [10224.253935] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10224.256745] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10224.259798] do_raw_spin_lock+0x177/0x190 [10224.262273] _raw_spin_lock+0x21/0x30 [10224.264755] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10224.267687] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10224.270642] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10224.273113] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10224.275642] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10224.278340] ? woken_wake_function+0x30/0x30 [10224.280175] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10224.283410] kthread+0x1d1/0x200 [10224.284541] ? set_kthread_struct+0x70/0x70 [10224.286674] ret_from_fork+0x1f/0x30 [10240.056448] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [10240.061378] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10240.062006] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [10240.062719] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [10240.062726] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10240.062804] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10240.062810] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10240.062816] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10240.063108] RIP: 0010:native_safe_halt+0x2e/0x50 [10240.063118] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10240.063123] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10240.063128] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10240.063131] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10240.063134] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10240.063136] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10240.063139] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10240.063142] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10240.063145] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10240.063147] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10240.063155] Call Trace: [10240.063159] [10240.063164] ? show_regs.cold.9+0x22/0x2f [10240.063172] ? watchdog_timer_fn.cold.6+0x62/0x10a [10240.063178] ? __hrtimer_run_queues+0x1c2/0x4d0 [10240.063182] ? watchdog+0x40/0x40 [10240.063186] ? hrtimer_interrupt+0x128/0x2c0 [10240.063190] ? smp_apic_timer_interrupt+0xa6/0x270 [10240.063196] ? apic_timer_interrupt+0xf/0x20 [10240.063201] [10240.063204] ? native_safe_halt+0x2e/0x50 [10240.063213] kvm_wait+0xcd/0xe0 [10240.063220] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10240.063227] do_raw_spin_lock+0x177/0x190 [10240.063231] _raw_spin_lock+0x21/0x30 [10240.063235] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10240.063473] process_one_work+0x2c8/0x700 [10240.063480] worker_thread+0x296/0x6e0 [10240.063484] ? rescuer_thread+0x570/0x570 [10240.063488] kthread+0x1d1/0x200 [10240.063492] ? set_kthread_struct+0x70/0x70 [10240.063495] ret_from_fork+0x1f/0x30 [10240.078217] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10240.078233] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10240.078236] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10240.078251] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10240.078256] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10240.078262] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10240.078265] RDX: 0000000000006d9e RSI: 0000000000000003 RDI: ffff9a61907912e0 [10240.078267] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10240.078269] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10240.078271] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10240.078274] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10240.078276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10240.078279] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10240.078287] Call Trace: [10240.078292] [10240.078298] ? show_regs.cold.9+0x22/0x2f [10240.078307] ? watchdog_timer_fn.cold.6+0x62/0x10a [10240.078314] ? __hrtimer_run_queues+0x1c2/0x4d0 [10240.078318] ? watchdog+0x40/0x40 [10240.078323] ? hrtimer_interrupt+0x128/0x2c0 [10240.078327] ? smp_apic_timer_interrupt+0xa6/0x270 [10240.078336] ? apic_timer_interrupt+0xf/0x20 [10240.078340] [10240.078405] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10240.078412] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10240.078417] do_raw_spin_lock+0x177/0x190 [10240.078422] _raw_spin_lock+0x21/0x30 [10240.078432] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10240.078956] obd_mod_free+0x46/0x140 [ptlrpc] [10240.079126] mdc_commit_open+0x91/0xb0 [mdc] [10240.079165] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10240.087698] Modules linked in: [10240.090389] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10240.147005] lustre(O) [10240.153951] after_reply+0xe2c/0x1780 [ptlrpc] [10240.154655] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10240.154969] ? __next_timer_interrupt+0x160/0x160 [10240.154983] ptlrpcd+0xa82/0xe40 [ptlrpc] [10240.155423] ? do_wait_intr+0xf0/0xf0 [10240.155433] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10240.155675] kthread+0x1d1/0x200 [10240.155680] ? set_kthread_struct+0x70/0x70 [10240.155683] ret_from_fork+0x1f/0x30 [10240.560739] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10240.596506] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10240.609584] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10240.620478] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10240.628700] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10240.645757] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [10240.649762] RAX: 0000000000004d2c RBX: ffff9a61907912e0 RCX: 0000000000000001 [10240.656375] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10240.659776] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10240.666121] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10240.670455] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10240.674808] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10240.680886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10240.685020] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10240.688084] Call Trace: [10240.688892] [10240.691118] ? show_regs.cold.9+0x22/0x2f [10240.692771] ? watchdog_timer_fn.cold.6+0x62/0x10a [10240.698328] ? __hrtimer_run_queues+0x1c2/0x4d0 [10240.700044] ? watchdog+0x40/0x40 [10240.701311] ? hrtimer_interrupt+0x128/0x2c0 [10240.704988] ? smp_apic_timer_interrupt+0xa6/0x270 [10240.707322] ? apic_timer_interrupt+0xf/0x20 [10240.711420] [10240.712268] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10240.715400] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10240.717763] do_raw_spin_lock+0x177/0x190 [10240.719722] _raw_spin_lock+0x21/0x30 [10240.721978] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10240.724799] ? do_raw_spin_unlock+0x75/0x190 [10240.727032] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10240.730176] ? do_wait_intr+0xf0/0xf0 [10240.731704] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10240.734797] mdc_reint+0x2d/0x110 [mdc] [10240.737000] mdc_unlink+0x658/0x870 [mdc] [10240.739240] lmv_unlink+0x34d/0xbd0 [lmv] [10240.741295] ll_unlink+0x24d/0x7f0 [lustre] [10240.744269] vfs_unlink+0x202/0x330 [10240.747050] do_unlinkat+0x2fa/0x460 [10240.749009] __x64_sys_unlink+0x37/0x50 [10240.752351] do_syscall_64+0xc1/0x440 [10240.754295] entry_SYSCALL_64_after_hwframe+0x49/0xae [10240.756880] RIP: 0033:0x7f83e5f5c10b [10240.758472] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10240.769204] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10240.773156] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10240.777236] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10240.780969] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10240.784801] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10240.788605] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10246.112163] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7732s! [10246.118293] Showing busy workqueues and worker pools: [10246.122791] workqueue events: flags=0x0 [10246.124908] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10246.128962] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10246.135200] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10246.139992] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10246.144528] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10246.149270] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10246.154439] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10246.158755] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10246.164567] workqueue events_long: flags=0x0 [10246.167281] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10246.172081] pending: lu_objects_flush [obdclass] [10246.175690] workqueue events_unbound: flags=0x2 [10246.178193] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10246.182111] pending: flush_memcg_stats_dwork, idle_cull_fn [10246.185940] workqueue events_power_efficient: flags=0x80 [10246.189280] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10246.193147] pending: neigh_periodic_work, do_cache_clean [10246.195860] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10246.199675] pending: neigh_periodic_work, check_lifetime [10246.203277] workqueue mm_percpu_wq: flags=0x8 [10246.206610] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10246.210850] pending: vmstat_update, lru_add_drain_per_cpu [10246.214696] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10246.218547] pending: vmstat_update [10246.221074] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10246.225298] pending: lru_add_drain_per_cpu BAR(48) [10246.228188] workqueue ipv6_addrconf: flags=0x40008 [10246.231037] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10246.234079] pending: addrconf_verify_work [10246.235865] workqueue knbd0-recv: flags=0x1a [10246.238246] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10246.242321] in-flight: 650:recv_work [10246.245091] workqueue ptlrpc_pinger: flags=0x2 [10246.247693] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10246.251671] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10246.256021] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7733s workers=4 idle: 36 466 29757 [10246.260899] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10236s workers=2 idle: 652 [10252.059047] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [10252.063312] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10252.082366] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10252.087055] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10252.091052] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10252.094517] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10252.105457] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10252.108686] RAX: 000000000000710d RBX: ffff9a61907912e0 RCX: 0000000000000001 [10252.111616] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10252.114205] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10252.116462] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10252.119044] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10252.122059] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10252.125785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10252.128692] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10252.131843] Call Trace: [10252.132988] [10252.133716] ? show_regs.cold.9+0x22/0x2f [10252.135522] ? watchdog_timer_fn.cold.6+0x62/0x10a [10252.137660] ? __hrtimer_run_queues+0x1c2/0x4d0 [10252.139677] ? watchdog+0x40/0x40 [10252.141184] ? hrtimer_interrupt+0x128/0x2c0 [10252.143656] ? smp_apic_timer_interrupt+0xa6/0x270 [10252.145531] ? apic_timer_interrupt+0xf/0x20 [10252.147041] [10252.147678] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10252.149464] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10252.151499] do_raw_spin_lock+0x177/0x190 [10252.152863] _raw_spin_lock+0x21/0x30 [10252.154535] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10252.157033] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10252.159025] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10252.161585] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10252.164343] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10252.166474] ? woken_wake_function+0x30/0x30 [10252.168088] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10252.170423] kthread+0x1d1/0x200 [10252.171527] ? set_kthread_struct+0x70/0x70 [10252.173236] ret_from_fork+0x1f/0x30 [10268.060011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [10268.062024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [10268.064410] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10268.064486] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10268.064493] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10268.064499] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10268.067750] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10268.067834] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10268.067842] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10268.067845] RIP: 0010:native_safe_halt+0x2e/0x50 [10268.067857] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10268.067861] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [10268.067866] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10268.067869] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10268.067871] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10268.067873] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10268.067878] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10268.067881] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10268.067884] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10268.067887] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10268.067894] Call Trace: [10268.067902] [10268.067907] ? show_regs.cold.9+0x22/0x2f [10268.067916] ? watchdog_timer_fn.cold.6+0x62/0x10a [10268.067922] ? __hrtimer_run_queues+0x1c2/0x4d0 [10268.067926] ? watchdog+0x40/0x40 [10268.067932] ? hrtimer_interrupt+0x128/0x2c0 [10268.067936] ? smp_apic_timer_interrupt+0xa6/0x270 [10268.067942] ? apic_timer_interrupt+0xf/0x20 [10268.067946] [10268.067949] ? native_safe_halt+0x2e/0x50 [10268.067952] kvm_wait+0xcd/0xe0 [10268.068030] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10268.068038] do_raw_spin_lock+0x177/0x190 [10268.068042] _raw_spin_lock+0x21/0x30 [10268.068047] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10268.068316] ? do_raw_spin_unlock+0x75/0x190 [10268.068321] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10268.068523] ? do_wait_intr+0xf0/0xf0 [10268.068530] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10268.068852] mdc_reint+0x2d/0x110 [mdc] [10268.068896] mdc_unlink+0x658/0x870 [mdc] [10268.069094] lmv_unlink+0x34d/0xbd0 [lmv] [10268.069126] ll_unlink+0x24d/0x7f0 [lustre] [10268.069440] vfs_unlink+0x202/0x330 [10268.069453] do_unlinkat+0x2fa/0x460 [10268.069457] __x64_sys_unlink+0x37/0x50 [10268.069460] do_syscall_64+0xc1/0x440 [10268.069467] entry_SYSCALL_64_after_hwframe+0x49/0xae [10268.069473] RIP: 0033:0x7f83e5f5c10b [10268.069481] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10268.069485] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10268.069490] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10268.069492] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10268.069494] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10268.069497] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10268.069499] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10268.411008] [10268.411955] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [10268.416271] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [10268.423926] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10268.427322] RAX: 0000000000003ec5 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10268.430957] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10268.433893] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10268.436926] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10268.440370] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10268.443681] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10268.447182] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10268.449696] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10268.452684] Call Trace: [10268.453803] [10268.454808] ? show_regs.cold.9+0x22/0x2f [10268.456607] ? watchdog_timer_fn.cold.6+0x62/0x10a [10268.458906] ? __hrtimer_run_queues+0x1c2/0x4d0 [10268.460964] ? watchdog+0x40/0x40 [10268.462440] ? hrtimer_interrupt+0x128/0x2c0 [10268.464397] ? smp_apic_timer_interrupt+0xa6/0x270 [10268.466018] ? apic_timer_interrupt+0xf/0x20 [10268.467226] [10268.468066] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [10268.470417] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10268.472920] do_raw_spin_lock+0x177/0x190 [10268.474739] _raw_spin_lock+0x21/0x30 [10268.476744] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10268.479919] process_one_work+0x2c8/0x700 [10268.482286] worker_thread+0x296/0x6e0 [10268.484750] ? rescuer_thread+0x570/0x570 [10268.487040] kthread+0x1d1/0x200 [10268.488335] ? set_kthread_struct+0x70/0x70 [10268.494074] ret_from_fork+0x1f/0x30 [10272.057102] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [10272.061143] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10272.083241] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10272.088150] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10272.092476] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10272.095007] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10272.103368] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10272.106820] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10272.109888] RDX: 0000000000006571 RSI: 0000000000000003 RDI: ffff9a61907912e0 [10272.112899] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10272.115723] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10272.118454] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10272.121478] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10272.124161] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10272.126193] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10272.129160] Call Trace: [10272.130263] [10272.131216] ? show_regs.cold.9+0x22/0x2f [10272.132719] ? watchdog_timer_fn.cold.6+0x62/0x10a [10272.134849] ? __hrtimer_run_queues+0x1c2/0x4d0 [10272.136852] ? watchdog+0x40/0x40 [10272.138758] ? hrtimer_interrupt+0x128/0x2c0 [10272.140899] ? smp_apic_timer_interrupt+0xa6/0x270 [10272.142963] ? apic_timer_interrupt+0xf/0x20 [10272.144868] [10272.145824] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10272.148047] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10272.149881] do_raw_spin_lock+0x177/0x190 [10272.151475] _raw_spin_lock+0x21/0x30 [10272.152860] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10272.154966] obd_mod_free+0x46/0x140 [ptlrpc] [10272.157100] mdc_commit_open+0x91/0xb0 [mdc] [10272.158742] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10272.161044] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10272.163061] after_reply+0xe2c/0x1780 [ptlrpc] [10272.167818] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10272.170377] ? __next_timer_interrupt+0x160/0x160 [10272.172004] ptlrpcd+0xa82/0xe40 [ptlrpc] [10272.175156] ? do_wait_intr+0xf0/0xf0 [10272.177900] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10272.181272] kthread+0x1d1/0x200 [10272.183503] ? set_kthread_struct+0x70/0x70 [10272.185292] ret_from_fork+0x1f/0x30 [10276.321130] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7763s! [10276.324900] Showing busy workqueues and worker pools: [10276.327319] workqueue events: flags=0x0 [10276.333017] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10276.339501] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10276.346388] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10276.350212] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10276.356115] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10276.361868] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10276.367321] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10276.374829] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10276.380870] workqueue events_long: flags=0x0 [10276.384404] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10276.392197] pending: lu_objects_flush [obdclass] [10276.396342] workqueue events_unbound: flags=0x2 [10276.399255] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10276.404742] pending: flush_memcg_stats_dwork, idle_cull_fn [10276.406826] workqueue events_power_efficient: flags=0x80 [10276.414957] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10276.422318] pending: neigh_periodic_work, do_cache_clean [10276.426190] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10276.437316] pending: neigh_periodic_work, check_lifetime [10276.442463] workqueue mm_percpu_wq: flags=0x8 [10276.447923] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10276.456292] pending: vmstat_update, lru_add_drain_per_cpu [10276.458464] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10276.463820] pending: vmstat_update [10276.464712] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10276.474291] pending: lru_add_drain_per_cpu BAR(48) [10276.478421] workqueue ipv6_addrconf: flags=0x40008 [10276.480074] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10276.483907] pending: addrconf_verify_work [10276.487441] workqueue knbd0-recv: flags=0x1a [10276.489584] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10276.492876] in-flight: 650:recv_work [10276.494899] workqueue ptlrpc_pinger: flags=0x2 [10276.500813] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10276.507012] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10276.509985] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7763s workers=4 idle: 36 466 29757 [10276.515831] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10266s workers=2 idle: 652 [10280.059091] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [10280.061627] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10280.078962] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10280.082944] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10280.086401] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10280.088930] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10280.095946] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10280.098567] RAX: 000000000000538e RBX: ffff9a61907912e0 RCX: 0000000000000001 [10280.101710] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10280.104837] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10280.107292] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10280.110079] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10280.112951] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10280.115650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10280.117909] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10280.120530] Call Trace: [10280.122965] [10280.123731] ? show_regs.cold.9+0x22/0x2f [10280.125111] ? watchdog_timer_fn.cold.6+0x62/0x10a [10280.128324] ? __hrtimer_run_queues+0x1c2/0x4d0 [10280.130483] ? watchdog+0x40/0x40 [10280.131593] ? hrtimer_interrupt+0x128/0x2c0 [10280.133239] ? smp_apic_timer_interrupt+0xa6/0x270 [10280.135380] ? apic_timer_interrupt+0xf/0x20 [10280.136872] [10280.137723] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10280.139495] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10280.141347] do_raw_spin_lock+0x177/0x190 [10280.142754] _raw_spin_lock+0x21/0x30 [10280.143946] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10280.146561] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10280.148925] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10280.151738] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10280.154185] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10280.156541] ? woken_wake_function+0x30/0x30 [10280.158541] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10280.161186] kthread+0x1d1/0x200 [10280.162262] ? set_kthread_struct+0x70/0x70 [10280.163728] ret_from_fork+0x1f/0x30 [10296.060010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [10296.062761] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10296.063909] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [10296.082042] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10296.090247] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash [10296.095931] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10296.095945] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10296.096272] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10296.096285] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10296.096289] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10296.096293] RAX: 0000000000002086 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10296.096296] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10296.096297] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10296.096299] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10296.096301] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10296.096304] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10296.096307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10296.096309] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10296.096318] Call Trace: [10296.096324] [10296.096329] ? show_regs.cold.9+0x22/0x2f [10296.096340] ? watchdog_timer_fn.cold.6+0x62/0x10a [10296.096346] ? __hrtimer_run_queues+0x1c2/0x4d0 [10296.096350] ? watchdog+0x40/0x40 [10296.096354] ? hrtimer_interrupt+0x128/0x2c0 [10296.096358] ? smp_apic_timer_interrupt+0xa6/0x270 [10296.096366] ? apic_timer_interrupt+0xf/0x20 [10296.096370] [10296.096372] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10296.096377] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10296.096381] do_raw_spin_lock+0x177/0x190 [10296.096385] _raw_spin_lock+0x21/0x30 [10296.096390] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10296.096599] process_one_work+0x2c8/0x700 [10296.096607] worker_thread+0x296/0x6e0 [10296.096612] ? rescuer_thread+0x570/0x570 [10296.096616] kthread+0x1d1/0x200 [10296.096621] ? set_kthread_struct+0x70/0x70 [10296.096624] ret_from_fork+0x1f/0x30 [10296.297742] dm_log dm_mod sha512_ssse3 sha512_generic [10296.302367] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10296.312917] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10296.323767] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10296.329777] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10296.341770] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10296.345878] RAX: 00000000000077ed RBX: ffff9a61907912e0 RCX: 0000000000000001 [10296.355185] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10296.362858] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10296.366639] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10296.371687] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10296.375668] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10296.381646] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10296.386189] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10296.392132] Call Trace: [10296.395907] [10296.398806] ? show_regs.cold.9+0x22/0x2f [10296.403922] ? watchdog_timer_fn.cold.6+0x62/0x10a [10296.408036] ? __hrtimer_run_queues+0x1c2/0x4d0 [10296.409848] ? watchdog+0x40/0x40 [10296.413679] ? hrtimer_interrupt+0x128/0x2c0 [10296.416142] ? smp_apic_timer_interrupt+0xa6/0x270 [10296.419156] ? apic_timer_interrupt+0xf/0x20 [10296.422000] [10296.423019] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10296.426600] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10296.430941] do_raw_spin_lock+0x177/0x190 [10296.434001] _raw_spin_lock+0x21/0x30 [10296.437697] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10296.446848] ? do_raw_spin_unlock+0x75/0x190 [10296.449839] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10296.458134] ? do_wait_intr+0xf0/0xf0 [10296.463963] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10296.469355] mdc_reint+0x2d/0x110 [mdc] [10296.470841] mdc_unlink+0x658/0x870 [mdc] [10296.478724] lmv_unlink+0x34d/0xbd0 [lmv] [10296.482653] ll_unlink+0x24d/0x7f0 [lustre] [10296.485132] vfs_unlink+0x202/0x330 [10296.488910] do_unlinkat+0x2fa/0x460 [10296.492651] __x64_sys_unlink+0x37/0x50 [10296.498974] do_syscall_64+0xc1/0x440 [10296.500426] entry_SYSCALL_64_after_hwframe+0x49/0xae [10296.510784] RIP: 0033:0x7f83e5f5c10b [10296.516211] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10296.542985] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10296.551262] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10296.561633] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10296.570984] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10296.577842] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10296.581971] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10304.056042] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [10304.061091] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10304.078080] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10304.083798] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10304.087599] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10304.090145] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10304.097727] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10304.100688] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10304.103478] RDX: 0000000000005825 RSI: 0000000000000003 RDI: ffff9a61907912e0 [10304.106291] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10304.109107] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10304.112050] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10304.115141] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10304.118536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10304.121242] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10304.124172] Call Trace: [10304.125008] [10304.125787] ? show_regs.cold.9+0x22/0x2f [10304.127764] ? watchdog_timer_fn.cold.6+0x62/0x10a [10304.130602] ? __hrtimer_run_queues+0x1c2/0x4d0 [10304.132783] ? watchdog+0x40/0x40 [10304.134666] ? hrtimer_interrupt+0x128/0x2c0 [10304.136747] ? smp_apic_timer_interrupt+0xa6/0x270 [10304.139492] ? apic_timer_interrupt+0xf/0x20 [10304.141715] [10304.142703] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10304.146442] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10304.149647] do_raw_spin_lock+0x177/0x190 [10304.151622] _raw_spin_lock+0x21/0x30 [10304.153596] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10304.156438] obd_mod_free+0x46/0x140 [ptlrpc] [10304.158523] mdc_commit_open+0x91/0xb0 [mdc] [10304.160648] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10304.163486] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10304.166142] after_reply+0xe2c/0x1780 [ptlrpc] [10304.169315] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10304.171248] ? __next_timer_interrupt+0x160/0x160 [10304.185353] ptlrpcd+0xa82/0xe40 [ptlrpc] [10304.187796] ? do_wait_intr+0xf0/0xf0 [10304.189025] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10304.191410] kthread+0x1d1/0x200 [10304.192596] ? set_kthread_struct+0x70/0x70 [10304.193866] ret_from_fork+0x1f/0x30 [10306.529203] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7793s! [10306.538569] Showing busy workqueues and worker pools: [10306.542695] workqueue events: flags=0x0 [10306.549557] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10306.559014] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10306.576628] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10306.578554] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10306.587627] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10306.594252] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10306.609120] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10306.613944] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10306.623622] workqueue events_long: flags=0x0 [10306.626546] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10306.636955] pending: lu_objects_flush [obdclass] [10306.642145] workqueue events_unbound: flags=0x2 [10306.644373] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10306.647764] pending: flush_memcg_stats_dwork, idle_cull_fn [10306.650324] workqueue events_power_efficient: flags=0x80 [10306.652503] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10306.655461] pending: neigh_periodic_work, do_cache_clean [10306.657866] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10306.660982] pending: neigh_periodic_work, check_lifetime [10306.663096] workqueue mm_percpu_wq: flags=0x8 [10306.664706] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10306.666998] pending: vmstat_update, lru_add_drain_per_cpu [10306.668791] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10306.671410] pending: vmstat_update [10306.672739] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10306.675129] pending: lru_add_drain_per_cpu BAR(48) [10306.676815] workqueue ipv6_addrconf: flags=0x40008 [10306.678495] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10306.680665] pending: addrconf_verify_work [10306.681916] workqueue knbd0-recv: flags=0x1a [10306.683690] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10306.686617] in-flight: 650:recv_work [10306.688114] workqueue ptlrpc_pinger: flags=0x2 [10306.689879] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10306.692839] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10306.696272] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7793s workers=4 idle: 36 466 29757 [10306.700507] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10296s workers=2 idle: 652 [10308.059349] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [10308.065526] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10308.114523] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10308.125265] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10308.130986] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10308.133729] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10308.146148] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10308.149684] RAX: 0000000000003892 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10308.153293] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10308.157555] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10308.162768] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10308.168464] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10308.177917] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10308.183458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10308.188459] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10308.195363] Call Trace: [10308.197251] [10308.197926] ? show_regs.cold.9+0x22/0x2f [10308.199266] ? watchdog_timer_fn.cold.6+0x62/0x10a [10308.208076] ? __hrtimer_run_queues+0x1c2/0x4d0 [10308.219005] ? watchdog+0x40/0x40 [10308.223458] ? hrtimer_interrupt+0x128/0x2c0 [10308.225827] ? smp_apic_timer_interrupt+0xa6/0x270 [10308.232382] ? apic_timer_interrupt+0xf/0x20 [10308.233936] [10308.242830] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10308.245090] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10308.249691] do_raw_spin_lock+0x177/0x190 [10308.252042] _raw_spin_lock+0x21/0x30 [10308.256035] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10308.262395] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10308.267040] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10308.272690] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10308.276762] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10308.283005] ? woken_wake_function+0x30/0x30 [10308.287807] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10308.291856] kthread+0x1d1/0x200 [10308.297518] ? set_kthread_struct+0x70/0x70 [10308.301522] ret_from_fork+0x1f/0x30 [10321.172723] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [10321.182644] rcu: 0-....: (5226581 ticks this GP) idle=30a/1/0x4000000000000000 softirq=128519/128519 fqs=126395 [10321.195284] (detected by 1, t=8340213 jiffies, g=262441, q=59741) [10321.199283] Sending NMI from CPU 1 to CPUs 0: [10321.201987] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [10321.203826] rcu: rcu_sched kthread starved for 7809400 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [10321.210609] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10321.224406] rcu: RCU grace-period kthread stack dump: [10321.226034] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10321.230758] Call Trace: [10321.231604] __schedule+0x351/0xcb0 [10321.232843] schedule+0xc0/0x180 [10321.234764] schedule_timeout+0xb4/0x190 [10321.248128] ? do_raw_spin_unlock+0x75/0x190 [10321.250801] ? __next_timer_interrupt+0x160/0x160 [10321.252762] ? prepare_to_swait_event+0xf3/0x140 [10321.257005] rcu_gp_kthread+0x8dd/0x15b0 [10321.260653] ? force_qs_rnp+0x2c0/0x2c0 [10321.263595] kthread+0x1d1/0x200 [10321.265265] ? set_kthread_struct+0x70/0x70 [10321.267139] ret_from_fork+0x1f/0x30 [10321.268683] rcu: Stack dump where RCU GP kthread last ran: [10321.273341] NMI backtrace for cpu 1 [10321.274663] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10321.283008] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10321.286875] Call Trace: [10321.287837] [10321.288609] ? dump_stack+0xbb/0x10e [10321.290955] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10321.293291] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10321.295372] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10321.297469] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10321.299593] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10321.304635] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [10321.307460] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [10321.311525] ? update_process_times+0x6d/0xc0 [10321.314979] ? tick_sched_handle.isra.18+0x27/0x90 [10321.318386] ? tick_sched_timer+0x53/0xb0 [10321.320319] ? __hrtimer_run_queues+0x1c2/0x4d0 [10321.322745] ? tick_sched_handle.isra.18+0x90/0x90 [10321.324583] ? hrtimer_interrupt+0x128/0x2c0 [10321.326592] ? smp_apic_timer_interrupt+0xa6/0x270 [10321.328250] ? apic_timer_interrupt+0xf/0x20 [10321.329600] [10321.331390] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10321.336006] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10321.342318] ? do_raw_spin_lock+0x177/0x190 [10321.343540] ? _raw_spin_lock+0x21/0x30 [10321.348825] ? ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10321.352565] ? ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10321.359976] ? ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10321.368055] ? ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10321.370774] ? ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10321.373666] ? woken_wake_function+0x30/0x30 [10321.376510] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10321.379540] ? kthread+0x1d1/0x200 [10321.385013] ? set_kthread_struct+0x70/0x70 [10321.386329] ? ret_from_fork+0x1f/0x30 [10324.062504] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [10324.063009] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [10324.065253] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10324.065325] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10324.065332] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10324.065338] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10324.070097] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10324.070232] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10324.070240] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10324.070243] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10324.091513] [10324.091521] RIP: 0010:native_safe_halt+0x2e/0x50 [10324.091542] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10324.091546] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10324.091551] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10324.091554] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10324.091556] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10324.091558] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10324.091560] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10324.091564] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10324.091566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10324.091569] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10324.091575] Call Trace: [10324.091580] [10324.091586] ? show_regs.cold.9+0x22/0x2f [10324.091594] ? watchdog_timer_fn.cold.6+0x62/0x10a [10324.091601] ? __hrtimer_run_queues+0x1c2/0x4d0 [10324.091605] ? watchdog+0x40/0x40 [10324.091609] ? hrtimer_interrupt+0x128/0x2c0 [10324.091613] ? smp_apic_timer_interrupt+0xa6/0x270 [10324.091618] ? apic_timer_interrupt+0xf/0x20 [10324.091623] [10324.091625] ? native_safe_halt+0x2e/0x50 [10324.091629] kvm_wait+0xcd/0xe0 [10324.091635] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10324.091643] do_raw_spin_lock+0x177/0x190 [10324.091647] _raw_spin_lock+0x21/0x30 [10324.091651] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10324.091925] process_one_work+0x2c8/0x700 [10324.091932] worker_thread+0x296/0x6e0 [10324.091936] ? rescuer_thread+0x570/0x570 [10324.091940] kthread+0x1d1/0x200 [10324.091944] ? set_kthread_struct+0x70/0x70 [10324.091947] ret_from_fork+0x1f/0x30 [10324.268505] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10324.278442] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10324.281831] RAX: 000000000000130a RBX: ffff9a61907912e0 RCX: 0000000000000001 [10324.286635] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10324.291650] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10324.294867] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10324.298979] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10324.304585] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10324.310274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10324.312606] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10324.316777] Call Trace: [10324.318298] [10324.319847] ? show_regs.cold.9+0x22/0x2f [10324.321378] ? watchdog_timer_fn.cold.6+0x62/0x10a [10324.327533] ? __hrtimer_run_queues+0x1c2/0x4d0 [10324.328740] ? watchdog+0x40/0x40 [10324.329773] ? hrtimer_interrupt+0x128/0x2c0 [10324.335219] ? smp_apic_timer_interrupt+0xa6/0x270 [10324.337218] ? apic_timer_interrupt+0xf/0x20 [10324.339445] [10324.340271] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10324.342852] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10324.348559] do_raw_spin_lock+0x177/0x190 [10324.350451] _raw_spin_lock+0x21/0x30 [10324.351854] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10324.355340] ? do_raw_spin_unlock+0x75/0x190 [10324.357736] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10324.361820] ? do_wait_intr+0xf0/0xf0 [10324.367393] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10324.369752] mdc_reint+0x2d/0x110 [mdc] [10324.370843] mdc_unlink+0x658/0x870 [mdc] [10324.371914] lmv_unlink+0x34d/0xbd0 [lmv] [10324.374826] ll_unlink+0x24d/0x7f0 [lustre] [10324.377223] vfs_unlink+0x202/0x330 [10324.379127] do_unlinkat+0x2fa/0x460 [10324.381902] __x64_sys_unlink+0x37/0x50 [10324.383255] do_syscall_64+0xc1/0x440 [10324.385328] entry_SYSCALL_64_after_hwframe+0x49/0xae [10324.386851] RIP: 0033:0x7f83e5f5c10b [10324.388761] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10324.403313] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10324.407487] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10324.412067] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10324.414985] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10324.423547] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10324.430257] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10336.737629] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7823s! [10336.743902] Showing busy workqueues and worker pools: [10336.750092] workqueue events: flags=0x0 [10336.757643] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10336.766641] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10336.777380] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10336.783291] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10336.786944] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10336.791018] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10336.795165] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10336.800115] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10336.808630] workqueue events_long: flags=0x0 [10336.812483] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10336.818528] pending: lu_objects_flush [obdclass] [10336.825109] workqueue events_unbound: flags=0x2 [10336.828570] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10336.833620] pending: flush_memcg_stats_dwork, idle_cull_fn [10336.837244] workqueue events_power_efficient: flags=0x80 [10336.841316] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10336.847409] pending: neigh_periodic_work, do_cache_clean [10336.853247] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10336.864376] pending: neigh_periodic_work, check_lifetime [10336.868748] workqueue mm_percpu_wq: flags=0x8 [10336.870082] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10336.876817] pending: vmstat_update, lru_add_drain_per_cpu [10336.886460] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10336.893464] pending: vmstat_update [10336.896631] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10336.905015] pending: lru_add_drain_per_cpu BAR(48) [10336.910474] workqueue ipv6_addrconf: flags=0x40008 [10336.917473] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10336.927893] pending: addrconf_verify_work [10336.929552] workqueue knbd0-recv: flags=0x1a [10336.931182] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10336.938460] in-flight: 650:recv_work [10336.943384] workqueue ptlrpc_pinger: flags=0x2 [10336.947164] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10336.960308] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10336.965626] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7823s workers=4 idle: 36 466 29757 [10336.973013] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10327s workers=2 idle: 652 [10348.059013] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [10348.062651] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10348.085742] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10348.091979] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10348.096356] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10348.099643] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10348.108833] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10348.112692] RAX: 0000000000001a4c RBX: ffff9a61907912e0 RCX: 0000000000000001 [10348.116355] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10348.120140] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10348.123473] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10348.127434] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10348.131268] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10348.135416] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10348.138321] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10348.142185] Call Trace: [10348.143698] [10348.144520] ? show_regs.cold.9+0x22/0x2f [10348.146949] ? watchdog_timer_fn.cold.6+0x62/0x10a [10348.149302] ? __hrtimer_run_queues+0x1c2/0x4d0 [10348.151737] ? watchdog+0x40/0x40 [10348.153079] ? hrtimer_interrupt+0x128/0x2c0 [10348.155636] ? smp_apic_timer_interrupt+0xa6/0x270 [10348.158580] ? apic_timer_interrupt+0xf/0x20 [10348.160994] [10348.162244] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10348.165360] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10348.168074] do_raw_spin_lock+0x177/0x190 [10348.170249] _raw_spin_lock+0x21/0x30 [10348.172325] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10348.175684] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10348.178516] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10348.181537] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10348.184397] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10348.186955] ? woken_wake_function+0x30/0x30 [10348.188598] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10348.192291] kthread+0x1d1/0x200 [10348.195433] ? set_kthread_struct+0x70/0x70 [10348.198673] ret_from_fork+0x1f/0x30 [10352.060074] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [10352.062640] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10352.066457] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [10352.104990] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10352.105004] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10352.105010] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10352.105353] RIP: 0010:native_safe_halt+0x2e/0x50 [10352.105364] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10352.105368] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10352.105373] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10352.105385] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10352.105388] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10352.105389] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10352.105392] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10352.105395] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10352.105398] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10352.105401] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10352.105409] Call Trace: [10352.105415] [10352.105422] ? show_regs.cold.9+0x22/0x2f [10352.105433] ? watchdog_timer_fn.cold.6+0x62/0x10a [10352.105439] ? __hrtimer_run_queues+0x1c2/0x4d0 [10352.105443] ? watchdog+0x40/0x40 [10352.105447] ? hrtimer_interrupt+0x128/0x2c0 [10352.105450] ? smp_apic_timer_interrupt+0xa6/0x270 [10352.105457] ? apic_timer_interrupt+0xf/0x20 [10352.105461] [10352.105464] ? native_safe_halt+0x2e/0x50 [10352.105468] kvm_wait+0xcd/0xe0 [10352.105475] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10352.105482] do_raw_spin_lock+0x177/0x190 [10352.105487] _raw_spin_lock+0x21/0x30 [10352.105491] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10352.105863] process_one_work+0x2c8/0x700 [10352.105871] worker_thread+0x296/0x6e0 [10352.105875] ? rescuer_thread+0x570/0x570 [10352.105879] kthread+0x1d1/0x200 [10352.105883] ? set_kthread_struct+0x70/0x70 [10352.105886] ret_from_fork+0x1f/0x30 [10352.317086] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10352.353073] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10352.359122] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10352.364915] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10352.369153] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10352.385713] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10352.390649] RAX: 0000000000005482 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10352.393989] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10352.396932] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10352.399623] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10352.402520] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10352.405184] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10352.408161] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10352.410429] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10352.413625] Call Trace: [10352.414606] [10352.415318] ? show_regs.cold.9+0x22/0x2f [10352.417430] ? watchdog_timer_fn.cold.6+0x62/0x10a [10352.419455] ? __hrtimer_run_queues+0x1c2/0x4d0 [10352.421351] ? watchdog+0x40/0x40 [10352.425386] ? hrtimer_interrupt+0x128/0x2c0 [10352.428488] ? smp_apic_timer_interrupt+0xa6/0x270 [10352.430958] ? apic_timer_interrupt+0xf/0x20 [10352.432652] [10352.433511] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10352.436309] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10352.439682] do_raw_spin_lock+0x177/0x190 [10352.442012] _raw_spin_lock+0x21/0x30 [10352.444639] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10352.447220] ? do_raw_spin_unlock+0x75/0x190 [10352.449223] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10352.453245] ? do_wait_intr+0xf0/0xf0 [10352.457594] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10352.460076] mdc_reint+0x2d/0x110 [mdc] [10352.463576] mdc_unlink+0x658/0x870 [mdc] [10352.465125] lmv_unlink+0x34d/0xbd0 [lmv] [10352.468038] ll_unlink+0x24d/0x7f0 [lustre] [10352.469638] vfs_unlink+0x202/0x330 [10352.471892] do_unlinkat+0x2fa/0x460 [10352.477073] __x64_sys_unlink+0x37/0x50 [10352.478867] do_syscall_64+0xc1/0x440 [10352.480376] entry_SYSCALL_64_after_hwframe+0x49/0xae [10352.484189] RIP: 0033:0x7f83e5f5c10b [10352.485336] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10352.492087] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10352.495408] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10352.502029] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10352.503874] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10352.509175] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10352.512659] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10364.056028] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [10364.059301] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10364.078472] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10364.083179] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10364.086542] RIP: 0010:native_safe_halt+0x2e/0x50 [10364.088525] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10364.096157] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10364.099220] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10364.102120] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [10364.105160] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10364.108756] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10364.112452] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10364.116013] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10364.120039] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10364.123061] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10364.126991] Call Trace: [10364.128485] [10364.129574] ? show_regs.cold.9+0x22/0x2f [10364.131793] ? watchdog_timer_fn.cold.6+0x62/0x10a [10364.133547] ? __hrtimer_run_queues+0x1c2/0x4d0 [10364.135017] ? watchdog+0x40/0x40 [10364.136267] ? hrtimer_interrupt+0x128/0x2c0 [10364.139482] ? smp_apic_timer_interrupt+0xa6/0x270 [10364.142456] ? apic_timer_interrupt+0xf/0x20 [10364.145263] [10364.146856] ? native_safe_halt+0x2e/0x50 [10364.149211] kvm_wait+0xcd/0xe0 [10364.150724] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10364.152204] do_raw_spin_lock+0x177/0x190 [10364.154621] _raw_spin_lock+0x21/0x30 [10364.155922] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10364.158959] obd_mod_free+0x46/0x140 [ptlrpc] [10364.161776] mdc_commit_open+0x91/0xb0 [mdc] [10364.163963] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10364.166554] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10364.169259] after_reply+0xe2c/0x1780 [ptlrpc] [10364.171943] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10364.174514] ? __next_timer_interrupt+0x160/0x160 [10364.176951] ptlrpcd+0xa82/0xe40 [ptlrpc] [10364.179177] ? do_wait_intr+0xf0/0xf0 [10364.181866] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10364.184016] kthread+0x1d1/0x200 [10364.186941] ? set_kthread_struct+0x70/0x70 [10364.191137] ret_from_fork+0x1f/0x30 [10367.456176] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7854s! [10367.478037] Showing busy workqueues and worker pools: [10367.484118] workqueue events: flags=0x0 [10367.488818] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10367.498198] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10367.508343] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10367.520427] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10367.535498] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10367.545883] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10367.560970] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10367.577568] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10367.587609] workqueue events_long: flags=0x0 [10367.596074] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10367.603457] pending: lu_objects_flush [obdclass] [10367.610127] workqueue events_unbound: flags=0x2 [10367.613366] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10367.621399] pending: flush_memcg_stats_dwork, idle_cull_fn [10367.625305] workqueue events_power_efficient: flags=0x80 [10367.628176] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10367.636239] pending: neigh_periodic_work, do_cache_clean [10367.638213] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10367.645836] pending: neigh_periodic_work, check_lifetime [10367.657659] workqueue mm_percpu_wq: flags=0x8 [10367.659192] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10367.665245] pending: vmstat_update, lru_add_drain_per_cpu [10367.667946] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10367.671336] pending: vmstat_update [10367.673090] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10367.680768] pending: lru_add_drain_per_cpu BAR(48) [10367.687588] workqueue ipv6_addrconf: flags=0x40008 [10367.690699] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10367.696351] pending: addrconf_verify_work [10367.699529] workqueue knbd0-recv: flags=0x1a [10367.705288] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10367.709468] in-flight: 650:recv_work [10367.713046] workqueue ptlrpc_pinger: flags=0x2 [10367.716653] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10367.724075] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10367.729469] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7854s workers=4 idle: 36 466 29757 [10367.735163] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10357s workers=2 idle: 652 [10376.059043] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [10376.061779] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10376.102174] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10376.110493] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10376.118484] RIP: 0010:native_safe_halt+0x2e/0x50 [10376.121253] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10376.136762] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10376.140327] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10376.143241] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10376.146006] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10376.153411] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10376.159788] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10376.167813] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10376.179875] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10376.186250] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10376.191143] Call Trace: [10376.194377] [10376.199374] ? show_regs.cold.9+0x22/0x2f [10376.202718] ? watchdog_timer_fn.cold.6+0x62/0x10a [10376.207440] ? __hrtimer_run_queues+0x1c2/0x4d0 [10376.210577] ? watchdog+0x40/0x40 [10376.212356] ? hrtimer_interrupt+0x128/0x2c0 [10376.215084] ? smp_apic_timer_interrupt+0xa6/0x270 [10376.218236] ? apic_timer_interrupt+0xf/0x20 [10376.220391] [10376.222316] ? native_safe_halt+0x2e/0x50 [10376.225434] kvm_wait+0xcd/0xe0 [10376.227421] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10376.230085] do_raw_spin_lock+0x177/0x190 [10376.235578] _raw_spin_lock+0x21/0x30 [10376.239369] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10376.244656] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10376.249376] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10376.252746] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10376.258030] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10376.263414] ? woken_wake_function+0x30/0x30 [10376.268594] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10376.273630] kthread+0x1d1/0x200 [10376.275400] ? set_kthread_struct+0x70/0x70 [10376.278757] ret_from_fork+0x1f/0x30 [10380.060115] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [10380.062010] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [10380.072988] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10380.073068] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10380.073075] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10380.073081] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10380.073363] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10380.073375] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10380.073379] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10380.073383] RAX: 0000000000005029 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10380.073386] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10380.073388] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10380.073390] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10380.073391] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10380.073394] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10380.073397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10380.073399] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10380.073408] Call Trace: [10380.073414] [10380.073419] ? show_regs.cold.9+0x22/0x2f [10380.073430] ? watchdog_timer_fn.cold.6+0x62/0x10a [10380.073437] ? __hrtimer_run_queues+0x1c2/0x4d0 [10380.073441] ? watchdog+0x40/0x40 [10380.073445] ? hrtimer_interrupt+0x128/0x2c0 [10380.073449] ? smp_apic_timer_interrupt+0xa6/0x270 [10380.073458] ? apic_timer_interrupt+0xf/0x20 [10380.073536] [10380.073539] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10380.073544] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10380.073550] do_raw_spin_lock+0x177/0x190 [10380.073555] _raw_spin_lock+0x21/0x30 [10380.073560] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10380.073766] process_one_work+0x2c8/0x700 [10380.073773] worker_thread+0x296/0x6e0 [10380.073778] ? rescuer_thread+0x570/0x570 [10380.073782] kthread+0x1d1/0x200 [10380.073786] ? set_kthread_struct+0x70/0x70 [10380.073790] ret_from_fork+0x1f/0x30 [10380.325786] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10380.376150] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10380.388082] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10380.393383] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10380.396668] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10380.410157] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10380.414623] RAX: 0000000000005d87 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10380.417723] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10380.422924] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10380.425074] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10380.435663] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10380.438485] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10380.441991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10380.448293] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10380.452347] Call Trace: [10380.453804] [10380.455190] ? show_regs.cold.9+0x22/0x2f [10380.458343] ? watchdog_timer_fn.cold.6+0x62/0x10a [10380.460350] ? __hrtimer_run_queues+0x1c2/0x4d0 [10380.461666] ? watchdog+0x40/0x40 [10380.464820] ? hrtimer_interrupt+0x128/0x2c0 [10380.467964] ? smp_apic_timer_interrupt+0xa6/0x270 [10380.470180] ? apic_timer_interrupt+0xf/0x20 [10380.473617] [10380.477159] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10380.480696] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10380.485869] do_raw_spin_lock+0x177/0x190 [10380.487452] _raw_spin_lock+0x21/0x30 [10380.490198] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10380.493101] ? do_raw_spin_unlock+0x75/0x190 [10380.504624] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10380.507819] ? do_wait_intr+0xf0/0xf0 [10380.514901] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10380.517941] mdc_reint+0x2d/0x110 [mdc] [10380.518970] mdc_unlink+0x658/0x870 [mdc] [10380.520077] lmv_unlink+0x34d/0xbd0 [lmv] [10380.532047] ll_unlink+0x24d/0x7f0 [lustre] [10380.534817] vfs_unlink+0x202/0x330 [10380.541097] do_unlinkat+0x2fa/0x460 [10380.543742] __x64_sys_unlink+0x37/0x50 [10380.548398] do_syscall_64+0xc1/0x440 [10380.551292] entry_SYSCALL_64_after_hwframe+0x49/0xae [10380.554268] RIP: 0033:0x7f83e5f5c10b [10380.563297] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10380.572739] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10380.580683] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10380.586345] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10380.598057] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10380.602407] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10380.607212] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10392.057408] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [10392.071836] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10392.121318] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10392.132554] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10392.147683] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10392.154373] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10392.180580] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10392.184593] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10392.189762] RDX: 00000000000070bc RSI: 0000000000000003 RDI: ffff9a61907912e0 [10392.194504] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10392.204720] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10392.211894] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10392.218292] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10392.223759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10392.229790] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10392.238574] Call Trace: [10392.245642] [10392.252914] ? show_regs.cold.9+0x22/0x2f [10392.260385] ? watchdog_timer_fn.cold.6+0x62/0x10a [10392.261749] ? __hrtimer_run_queues+0x1c2/0x4d0 [10392.266030] ? watchdog+0x40/0x40 [10392.267999] ? hrtimer_interrupt+0x128/0x2c0 [10392.275978] ? smp_apic_timer_interrupt+0xa6/0x270 [10392.280620] ? apic_timer_interrupt+0xf/0x20 [10392.282672] [10392.283962] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10392.287271] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10392.300268] do_raw_spin_lock+0x177/0x190 [10392.301793] _raw_spin_lock+0x21/0x30 [10392.310969] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10392.318167] obd_mod_free+0x46/0x140 [ptlrpc] [10392.321759] mdc_commit_open+0x91/0xb0 [mdc] [10392.326347] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10392.333502] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10392.341168] after_reply+0xe2c/0x1780 [ptlrpc] [10392.342966] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10392.347805] ? __next_timer_interrupt+0x160/0x160 [10392.354629] ptlrpcd+0xa82/0xe40 [ptlrpc] [10392.361277] ? do_wait_intr+0xf0/0xf0 [10392.363215] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10392.375871] kthread+0x1d1/0x200 [10392.378715] ? set_kthread_struct+0x70/0x70 [10392.385407] ret_from_fork+0x1f/0x30 [10398.177075] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7884s! [10398.181970] Showing busy workqueues and worker pools: [10398.185853] workqueue events: flags=0x0 [10398.188124] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10398.191791] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10398.199689] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10398.206272] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10398.210069] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10398.216901] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10398.223760] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10398.227182] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10398.233939] workqueue events_long: flags=0x0 [10398.239237] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10398.242857] pending: lu_objects_flush [obdclass] [10398.245597] workqueue events_unbound: flags=0x2 [10398.247361] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10398.250887] pending: flush_memcg_stats_dwork, idle_cull_fn [10398.255609] workqueue events_power_efficient: flags=0x80 [10398.258535] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10398.261962] pending: neigh_periodic_work, do_cache_clean [10398.265526] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10398.269501] pending: neigh_periodic_work, check_lifetime [10398.272623] workqueue mm_percpu_wq: flags=0x8 [10398.274727] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10398.278712] pending: vmstat_update, lru_add_drain_per_cpu [10398.281988] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10398.285761] pending: vmstat_update [10398.287909] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10398.293900] pending: lru_add_drain_per_cpu BAR(48) [10398.297266] workqueue ipv6_addrconf: flags=0x40008 [10398.301380] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10398.305837] pending: addrconf_verify_work [10398.308152] workqueue knbd0-recv: flags=0x1a [10398.310719] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10398.314296] in-flight: 650:recv_work [10398.316680] workqueue ptlrpc_pinger: flags=0x2 [10398.319140] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10398.324680] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10398.330840] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7885s workers=4 idle: 36 466 29757 [10398.338699] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10388s workers=2 idle: 652 [10404.062624] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [10404.072102] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10404.108786] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10404.114220] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10404.118601] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10404.133210] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10404.148654] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10404.158660] RAX: 0000000000007f19 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10404.164867] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10404.174540] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10404.191688] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10404.196526] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10404.199574] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10404.204581] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10404.214489] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10404.220118] Call Trace: [10404.221528] [10404.223399] ? show_regs.cold.9+0x22/0x2f [10404.225248] ? watchdog_timer_fn.cold.6+0x62/0x10a [10404.227972] ? __hrtimer_run_queues+0x1c2/0x4d0 [10404.230433] ? watchdog+0x40/0x40 [10404.231842] ? hrtimer_interrupt+0x128/0x2c0 [10404.235068] ? smp_apic_timer_interrupt+0xa6/0x270 [10404.241474] ? apic_timer_interrupt+0xf/0x20 [10404.248795] [10404.250837] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10404.254176] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10404.261441] do_raw_spin_lock+0x177/0x190 [10404.264205] _raw_spin_lock+0x21/0x30 [10404.266491] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10404.281283] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10404.288159] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10404.291476] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10404.296093] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10404.300687] ? woken_wake_function+0x30/0x30 [10404.302916] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10404.307801] kthread+0x1d1/0x200 [10404.319570] ? set_kthread_struct+0x70/0x70 [10404.323791] ret_from_fork+0x1f/0x30 [10408.060184] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [10408.062058] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [10408.069657] Modules linked in: [10408.073431] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10408.073511] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10408.073518] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10408.073521] RIP: 0010:native_safe_halt+0x2e/0x50 [10408.073535] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10408.073539] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [10408.073544] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10408.073546] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10408.073548] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10408.073550] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10408.073552] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10408.073556] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10408.073559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10408.073562] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10408.073572] Call Trace: [10408.073578] [10408.073583] ? show_regs.cold.9+0x22/0x2f [10408.073593] ? watchdog_timer_fn.cold.6+0x62/0x10a [10408.073599] ? __hrtimer_run_queues+0x1c2/0x4d0 [10408.073603] ? watchdog+0x40/0x40 [10408.073608] ? hrtimer_interrupt+0x128/0x2c0 [10408.073612] ? smp_apic_timer_interrupt+0xa6/0x270 [10408.073621] ? apic_timer_interrupt+0xf/0x20 [10408.073626] [10408.073629] ? native_safe_halt+0x2e/0x50 [10408.073633] kvm_wait+0xcd/0xe0 [10408.073641] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10408.073649] do_raw_spin_lock+0x177/0x190 [10408.073653] _raw_spin_lock+0x21/0x30 [10408.073658] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10408.074175] ? do_raw_spin_unlock+0x75/0x190 [10408.074183] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10408.074528] ? do_wait_intr+0xf0/0xf0 [10408.074536] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10408.074760] mdc_reint+0x2d/0x110 [mdc] [10408.074801] mdc_unlink+0x658/0x870 [mdc] [10408.074828] lmv_unlink+0x34d/0xbd0 [lmv] [10408.074865] ll_unlink+0x24d/0x7f0 [lustre] [10408.075124] vfs_unlink+0x202/0x330 [10408.075133] do_unlinkat+0x2fa/0x460 [10408.075139] __x64_sys_unlink+0x37/0x50 [10408.075143] do_syscall_64+0xc1/0x440 [10408.075151] entry_SYSCALL_64_after_hwframe+0x49/0xae [10408.075160] RIP: 0033:0x7f83e5f5c10b [10408.075168] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10408.075172] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10408.075177] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10408.075180] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10408.075182] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10408.075184] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10408.075186] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10408.225617] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10408.250591] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10408.256346] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10408.260878] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10408.265376] RIP: 0010:native_safe_halt+0x2e/0x50 [10408.268107] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10408.277284] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10408.280945] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10408.283254] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10408.286503] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10408.289267] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10408.293015] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10408.297140] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10408.302580] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10408.305464] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10408.309164] Call Trace: [10408.310465] [10408.311651] ? show_regs.cold.9+0x22/0x2f [10408.313918] ? watchdog_timer_fn.cold.6+0x62/0x10a [10408.316901] ? __hrtimer_run_queues+0x1c2/0x4d0 [10408.319119] ? watchdog+0x40/0x40 [10408.320556] ? hrtimer_interrupt+0x128/0x2c0 [10408.322537] ? smp_apic_timer_interrupt+0xa6/0x270 [10408.324977] ? apic_timer_interrupt+0xf/0x20 [10408.327115] [10408.327631] ? native_safe_halt+0x2e/0x50 [10408.329199] kvm_wait+0xcd/0xe0 [10408.330576] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10408.332463] do_raw_spin_lock+0x177/0x190 [10408.333627] _raw_spin_lock+0x21/0x30 [10408.335120] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10408.337327] process_one_work+0x2c8/0x700 [10408.338649] worker_thread+0x296/0x6e0 [10408.339850] ? rescuer_thread+0x570/0x570 [10408.341284] kthread+0x1d1/0x200 [10408.342191] ? set_kthread_struct+0x70/0x70 [10408.343466] ret_from_fork+0x1f/0x30 [10424.056029] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [10424.059124] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10424.082315] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10424.087669] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10424.091462] RIP: 0010:native_safe_halt+0x2e/0x50 [10424.093754] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10424.103464] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10424.107334] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10424.110971] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [10424.114564] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10424.118469] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10424.121718] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10424.125208] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10424.129125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10424.132291] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10424.135296] Call Trace: [10424.136299] [10424.136980] ? show_regs.cold.9+0x22/0x2f [10424.138961] ? watchdog_timer_fn.cold.6+0x62/0x10a [10424.140783] ? __hrtimer_run_queues+0x1c2/0x4d0 [10424.142877] ? watchdog+0x40/0x40 [10424.144459] ? hrtimer_interrupt+0x128/0x2c0 [10424.146542] ? smp_apic_timer_interrupt+0xa6/0x270 [10424.148744] ? apic_timer_interrupt+0xf/0x20 [10424.150457] [10424.151146] ? native_safe_halt+0x2e/0x50 [10424.153184] kvm_wait+0xcd/0xe0 [10424.154426] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10424.155956] do_raw_spin_lock+0x177/0x190 [10424.157733] _raw_spin_lock+0x21/0x30 [10424.159408] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10424.161990] obd_mod_free+0x46/0x140 [ptlrpc] [10424.164226] mdc_commit_open+0x91/0xb0 [mdc] [10424.166786] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10424.169313] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10424.172342] after_reply+0xe2c/0x1780 [ptlrpc] [10424.174949] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10424.177920] ? __next_timer_interrupt+0x160/0x160 [10424.180146] ptlrpcd+0xa82/0xe40 [ptlrpc] [10424.182179] ? do_wait_intr+0xf0/0xf0 [10424.183757] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10424.186291] kthread+0x1d1/0x200 [10424.187871] ? set_kthread_struct+0x70/0x70 [10424.189360] ret_from_fork+0x1f/0x30 [10428.385136] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7915s! [10428.388032] Showing busy workqueues and worker pools: [10428.390143] workqueue events: flags=0x0 [10428.391368] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10428.393501] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10428.396961] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10428.399107] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10428.400862] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10428.402999] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10428.405014] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10428.407382] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10428.409951] workqueue events_long: flags=0x0 [10428.411224] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10428.413471] pending: lu_objects_flush [obdclass] [10428.414988] workqueue events_unbound: flags=0x2 [10428.416580] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10428.418369] pending: flush_memcg_stats_dwork, idle_cull_fn [10428.419926] workqueue events_power_efficient: flags=0x80 [10428.421522] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10428.423348] pending: neigh_periodic_work, do_cache_clean [10428.425407] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10428.428057] pending: neigh_periodic_work, check_lifetime [10428.429616] workqueue mm_percpu_wq: flags=0x8 [10428.431098] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10428.433311] pending: vmstat_update, lru_add_drain_per_cpu [10428.436216] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10428.438611] pending: vmstat_update [10428.439967] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10428.441941] pending: lru_add_drain_per_cpu BAR(48) [10428.443590] workqueue ipv6_addrconf: flags=0x40008 [10428.445139] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10428.447481] pending: addrconf_verify_work [10428.448800] workqueue knbd0-recv: flags=0x1a [10428.450488] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10428.452782] in-flight: 650:recv_work [10428.454444] workqueue ptlrpc_pinger: flags=0x2 [10428.455767] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10428.457709] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10428.459911] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7915s workers=4 idle: 36 466 29757 [10428.462709] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10418s workers=2 idle: 652 [10432.063636] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [10432.079893] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10432.148552] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10432.158599] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10432.166879] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10432.174783] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10432.200266] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10432.206849] RAX: 0000000000005e4d RBX: ffff9a61907912e0 RCX: 0000000000000001 [10432.215681] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10432.226507] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10432.235346] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10432.242077] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10432.256062] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10432.264394] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10432.272706] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10432.277446] Call Trace: [10432.278827] [10432.283685] ? show_regs.cold.9+0x22/0x2f [10432.287469] ? watchdog_timer_fn.cold.6+0x62/0x10a [10432.289688] ? __hrtimer_run_queues+0x1c2/0x4d0 [10432.291520] ? watchdog+0x40/0x40 [10432.292685] ? hrtimer_interrupt+0x128/0x2c0 [10432.295546] ? smp_apic_timer_interrupt+0xa6/0x270 [10432.300914] ? apic_timer_interrupt+0xf/0x20 [10432.306503] [10432.308959] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10432.313948] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10432.319554] do_raw_spin_lock+0x177/0x190 [10432.322797] _raw_spin_lock+0x21/0x30 [10432.325390] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10432.329921] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10432.335085] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10432.342409] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10432.348651] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10432.355675] ? woken_wake_function+0x30/0x30 [10432.357912] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10432.368732] kthread+0x1d1/0x200 [10432.372053] ? set_kthread_struct+0x70/0x70 [10432.374629] ret_from_fork+0x1f/0x30 [10436.060021] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [10436.062635] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [10436.063420] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10436.063509] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10436.063516] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10436.063523] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10436.063860] RIP: 0010:native_safe_halt+0x2e/0x50 [10436.063874] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10436.063878] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [10436.063882] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10436.063884] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10436.063887] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10436.063889] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10436.063891] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10436.063899] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10436.063902] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10436.063904] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10436.063913] Call Trace: [10436.063918] [10436.063924] ? show_regs.cold.9+0x22/0x2f [10436.063933] ? watchdog_timer_fn.cold.6+0x62/0x10a [10436.063939] ? __hrtimer_run_queues+0x1c2/0x4d0 [10436.063943] ? watchdog+0x40/0x40 [10436.063949] ? hrtimer_interrupt+0x128/0x2c0 [10436.063953] ? smp_apic_timer_interrupt+0xa6/0x270 [10436.063960] ? apic_timer_interrupt+0xf/0x20 [10436.063963] [10436.063966] ? native_safe_halt+0x2e/0x50 [10436.063970] kvm_wait+0xcd/0xe0 [10436.063978] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10436.063986] do_raw_spin_lock+0x177/0x190 [10436.063990] _raw_spin_lock+0x21/0x30 [10436.063995] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10436.064330] process_one_work+0x2c8/0x700 [10436.064340] worker_thread+0x296/0x6e0 [10436.064344] ? rescuer_thread+0x570/0x570 [10436.064348] kthread+0x1d1/0x200 [10436.064353] ? set_kthread_struct+0x70/0x70 [10436.064357] ret_from_fork+0x1f/0x30 [10436.364591] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10436.404216] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10436.416091] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10436.423698] RIP: 0010:native_safe_halt+0x2e/0x50 [10436.428328] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10436.454758] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10436.463298] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10436.470295] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10436.473934] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10436.479305] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10436.487455] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10436.498226] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10436.507019] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10436.513840] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10436.520201] Call Trace: [10436.521926] [10436.523378] ? show_regs.cold.9+0x22/0x2f [10436.527098] ? watchdog_timer_fn.cold.6+0x62/0x10a [10436.529172] ? __hrtimer_run_queues+0x1c2/0x4d0 [10436.532596] ? watchdog+0x40/0x40 [10436.534305] ? hrtimer_interrupt+0x128/0x2c0 [10436.538057] ? smp_apic_timer_interrupt+0xa6/0x270 [10436.541551] ? apic_timer_interrupt+0xf/0x20 [10436.550303] [10436.552045] ? native_safe_halt+0x2e/0x50 [10436.555533] kvm_wait+0xcd/0xe0 [10436.557841] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10436.564807] do_raw_spin_lock+0x177/0x190 [10436.566270] _raw_spin_lock+0x21/0x30 [10436.572740] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10436.580058] ? do_raw_spin_unlock+0x75/0x190 [10436.582300] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10436.589106] ? do_wait_intr+0xf0/0xf0 [10436.592067] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10436.600560] mdc_reint+0x2d/0x110 [mdc] [10436.605889] mdc_unlink+0x658/0x870 [mdc] [10436.608844] lmv_unlink+0x34d/0xbd0 [lmv] [10436.612680] ll_unlink+0x24d/0x7f0 [lustre] [10436.618210] vfs_unlink+0x202/0x330 [10436.621393] do_unlinkat+0x2fa/0x460 [10436.624053] __x64_sys_unlink+0x37/0x50 [10436.630860] do_syscall_64+0xc1/0x440 [10436.635520] entry_SYSCALL_64_after_hwframe+0x49/0xae [10436.639925] RIP: 0033:0x7f83e5f5c10b [10436.642432] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10436.656252] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10436.664855] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10436.668825] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10436.671661] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10436.675492] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10436.690038] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10456.056027] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [10456.058820] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10456.078900] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10456.083677] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10456.088170] RIP: 0010:native_safe_halt+0x2e/0x50 [10456.090104] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10456.098364] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10456.101823] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10456.105142] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [10456.108436] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10456.111679] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10456.114986] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10456.118224] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10456.121828] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10456.124288] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10456.127988] Call Trace: [10456.129635] [10456.130923] ? show_regs.cold.9+0x22/0x2f [10456.132811] ? watchdog_timer_fn.cold.6+0x62/0x10a [10456.134617] ? __hrtimer_run_queues+0x1c2/0x4d0 [10456.136787] ? watchdog+0x40/0x40 [10456.137971] ? hrtimer_interrupt+0x128/0x2c0 [10456.139796] ? smp_apic_timer_interrupt+0xa6/0x270 [10456.142009] ? apic_timer_interrupt+0xf/0x20 [10456.143865] [10456.145027] ? native_safe_halt+0x2e/0x50 [10456.146666] kvm_wait+0xcd/0xe0 [10456.147959] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10456.150314] do_raw_spin_lock+0x177/0x190 [10456.152177] _raw_spin_lock+0x21/0x30 [10456.153825] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10456.156301] obd_mod_free+0x46/0x140 [ptlrpc] [10456.158711] mdc_commit_open+0x91/0xb0 [mdc] [10456.160589] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10456.162827] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10456.165556] after_reply+0xe2c/0x1780 [ptlrpc] [10456.167819] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10456.170404] ? __next_timer_interrupt+0x160/0x160 [10456.172770] ptlrpcd+0xa82/0xe40 [ptlrpc] [10456.174561] ? do_wait_intr+0xf0/0xf0 [10456.176393] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10456.178876] kthread+0x1d1/0x200 [10456.180490] ? set_kthread_struct+0x70/0x70 [10456.182110] ret_from_fork+0x1f/0x30 [10458.593043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7945s! [10458.598534] Showing busy workqueues and worker pools: [10458.600527] workqueue events: flags=0x0 [10458.603186] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10458.606243] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10458.618167] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10458.630208] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10458.639613] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10458.645322] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10458.650584] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10458.654148] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10458.661168] workqueue events_long: flags=0x0 [10458.669196] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10458.679241] pending: lu_objects_flush [obdclass] [10458.686409] workqueue events_unbound: flags=0x2 [10458.691572] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10458.700720] pending: flush_memcg_stats_dwork, idle_cull_fn [10458.706906] workqueue events_power_efficient: flags=0x80 [10458.714921] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10458.724675] pending: neigh_periodic_work, do_cache_clean [10458.729021] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10458.736035] pending: neigh_periodic_work, check_lifetime [10458.739165] workqueue mm_percpu_wq: flags=0x8 [10458.741359] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10458.750406] pending: vmstat_update, lru_add_drain_per_cpu [10458.755457] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10458.760734] pending: vmstat_update [10458.762918] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10458.772441] pending: lru_add_drain_per_cpu BAR(48) [10458.775653] workqueue ipv6_addrconf: flags=0x40008 [10458.778684] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10458.784936] pending: addrconf_verify_work [10458.788456] workqueue knbd0-recv: flags=0x1a [10458.795325] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10458.800100] in-flight: 650:recv_work [10458.803110] workqueue ptlrpc_pinger: flags=0x2 [10458.805889] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10458.810182] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10458.813120] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7945s workers=4 idle: 36 466 29757 [10458.816643] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10448s workers=2 idle: 652 [10460.059533] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [10460.063677] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10460.115593] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10460.125253] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10460.146093] RIP: 0010:native_safe_halt+0x2e/0x50 [10460.152422] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10460.174976] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10460.186298] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10460.197147] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10460.199941] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10460.211265] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10460.227069] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10460.236542] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10460.249130] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10460.257556] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10460.263671] Call Trace: [10460.264537] [10460.265241] ? show_regs.cold.9+0x22/0x2f [10460.266439] ? watchdog_timer_fn.cold.6+0x62/0x10a [10460.276117] ? __hrtimer_run_queues+0x1c2/0x4d0 [10460.278557] ? watchdog+0x40/0x40 [10460.281126] ? hrtimer_interrupt+0x128/0x2c0 [10460.283466] ? smp_apic_timer_interrupt+0xa6/0x270 [10460.288070] ? apic_timer_interrupt+0xf/0x20 [10460.290599] [10460.292000] ? native_safe_halt+0x2e/0x50 [10460.294068] kvm_wait+0xcd/0xe0 [10460.296451] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10460.300154] do_raw_spin_lock+0x177/0x190 [10460.304029] _raw_spin_lock+0x21/0x30 [10460.305879] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10460.309138] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10460.312316] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10460.315350] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10460.318286] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10460.322250] ? woken_wake_function+0x30/0x30 [10460.324548] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10460.329357] kthread+0x1d1/0x200 [10460.331635] ? set_kthread_struct+0x70/0x70 [10460.336741] ret_from_fork+0x1f/0x30 [10464.060025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [10464.062021] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [10464.066259] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10464.066336] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10464.066343] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10464.066349] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10464.071192] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10464.071273] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10464.071282] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10464.071285] RIP: 0010:native_safe_halt+0x2e/0x50 [10464.071298] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10464.071303] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10464.071307] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10464.071310] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10464.071312] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10464.071314] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10464.071317] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10464.071320] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10464.071324] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10464.071326] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10464.071334] Call Trace: [10464.071341] [10464.071348] ? show_regs.cold.9+0x22/0x2f [10464.071360] ? watchdog_timer_fn.cold.6+0x62/0x10a [10464.071366] ? __hrtimer_run_queues+0x1c2/0x4d0 [10464.071371] ? watchdog+0x40/0x40 [10464.071375] ? hrtimer_interrupt+0x128/0x2c0 [10464.071379] ? smp_apic_timer_interrupt+0xa6/0x270 [10464.071386] ? apic_timer_interrupt+0xf/0x20 [10464.071391] [10464.071394] ? native_safe_halt+0x2e/0x50 [10464.071399] kvm_wait+0xcd/0xe0 [10464.071407] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10464.071413] do_raw_spin_lock+0x177/0x190 [10464.071417] _raw_spin_lock+0x21/0x30 [10464.071422] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10464.071916] ? do_raw_spin_unlock+0x75/0x190 [10464.071923] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10464.072117] ? do_wait_intr+0xf0/0xf0 [10464.072125] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10464.109057] [10464.109067] RIP: 0010:native_safe_halt+0x2e/0x50 [10464.109085] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10464.109089] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10464.109094] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10464.109097] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10464.109099] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10464.109102] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10464.109105] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10464.109108] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10464.109111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10464.109113] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10464.109120] Call Trace: [10464.109125] [10464.109130] ? show_regs.cold.9+0x22/0x2f [10464.109137] ? watchdog_timer_fn.cold.6+0x62/0x10a [10464.109142] ? __hrtimer_run_queues+0x1c2/0x4d0 [10464.109145] ? watchdog+0x40/0x40 [10464.109148] ? hrtimer_interrupt+0x128/0x2c0 [10464.109150] ? smp_apic_timer_interrupt+0xa6/0x270 [10464.109155] ? apic_timer_interrupt+0xf/0x20 [10464.109159] [10464.109161] ? native_safe_halt+0x2e/0x50 [10464.109166] kvm_wait+0xcd/0xe0 [10464.109173] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10464.109182] do_raw_spin_lock+0x177/0x190 [10464.109187] _raw_spin_lock+0x21/0x30 [10464.109191] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10464.109480] process_one_work+0x2c8/0x700 [10464.109760] worker_thread+0x296/0x6e0 [10464.109764] ? rescuer_thread+0x570/0x570 [10464.109768] kthread+0x1d1/0x200 [10464.109772] ? set_kthread_struct+0x70/0x70 [10464.109775] ret_from_fork+0x1f/0x30 [10464.487208] mdc_reint+0x2d/0x110 [mdc] [10464.489261] mdc_unlink+0x658/0x870 [mdc] [10464.491249] lmv_unlink+0x34d/0xbd0 [lmv] [10464.493140] ll_unlink+0x24d/0x7f0 [lustre] [10464.495026] vfs_unlink+0x202/0x330 [10464.496239] do_unlinkat+0x2fa/0x460 [10464.497922] __x64_sys_unlink+0x37/0x50 [10464.499453] do_syscall_64+0xc1/0x440 [10464.501141] entry_SYSCALL_64_after_hwframe+0x49/0xae [10464.503710] RIP: 0033:0x7f83e5f5c10b [10464.505463] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10464.514471] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10464.517823] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10464.523409] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10464.527289] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10464.532628] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10464.536219] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10484.056056] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [10484.065706] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10484.106081] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10484.110314] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10484.115244] RIP: 0010:native_safe_halt+0x2e/0x50 [10484.124848] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10484.146704] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10484.156419] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10484.160765] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [10484.167930] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10484.174342] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10484.181558] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10484.184726] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10484.187733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10484.190402] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10484.200122] Call Trace: [10484.201652] [10484.202678] ? show_regs.cold.9+0x22/0x2f [10484.206318] ? watchdog_timer_fn.cold.6+0x62/0x10a [10484.209371] ? __hrtimer_run_queues+0x1c2/0x4d0 [10484.211736] ? watchdog+0x40/0x40 [10484.214262] ? hrtimer_interrupt+0x128/0x2c0 [10484.215850] ? smp_apic_timer_interrupt+0xa6/0x270 [10484.219127] ? apic_timer_interrupt+0xf/0x20 [10484.220541] [10484.221324] ? native_safe_halt+0x2e/0x50 [10484.231418] kvm_wait+0xcd/0xe0 [10484.233131] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10484.235950] do_raw_spin_lock+0x177/0x190 [10484.238183] _raw_spin_lock+0x21/0x30 [10484.240223] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10484.242832] obd_mod_free+0x46/0x140 [ptlrpc] [10484.247115] mdc_commit_open+0x91/0xb0 [mdc] [10484.251503] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10484.255397] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10484.258884] after_reply+0xe2c/0x1780 [ptlrpc] [10484.261172] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10484.266802] ? __next_timer_interrupt+0x160/0x160 [10484.269084] ptlrpcd+0xa82/0xe40 [ptlrpc] [10484.271592] ? do_wait_intr+0xf0/0xf0 [10484.273511] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10484.276927] kthread+0x1d1/0x200 [10484.278700] ? set_kthread_struct+0x70/0x70 [10484.285810] ret_from_fork+0x1f/0x30 [10488.059014] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [10488.063630] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10488.089803] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10488.096306] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10488.100953] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [10488.104479] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [10488.114609] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10488.117782] RAX: 00000000000044a5 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10488.123747] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10488.128220] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10488.131573] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10488.135086] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10488.138938] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10488.143026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10488.145900] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10488.149029] Call Trace: [10488.150335] [10488.151059] ? show_regs.cold.9+0x22/0x2f [10488.152821] ? watchdog_timer_fn.cold.6+0x62/0x10a [10488.155378] ? __hrtimer_run_queues+0x1c2/0x4d0 [10488.157043] ? watchdog+0x40/0x40 [10488.158464] ? hrtimer_interrupt+0x128/0x2c0 [10488.160559] ? smp_apic_timer_interrupt+0xa6/0x270 [10488.162865] ? apic_timer_interrupt+0xf/0x20 [10488.164801] [10488.165805] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [10488.168686] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10488.171545] do_raw_spin_lock+0x177/0x190 [10488.173950] _raw_spin_lock+0x21/0x30 [10488.175912] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10488.179072] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10488.181642] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10488.184091] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10488.187532] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10488.190176] ? woken_wake_function+0x30/0x30 [10488.192406] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10488.195549] kthread+0x1d1/0x200 [10488.197582] ? set_kthread_struct+0x70/0x70 [10488.199167] ret_from_fork+0x1f/0x30 [10489.321360] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7976s! [10489.325368] Showing busy workqueues and worker pools: [10489.328414] workqueue events: flags=0x0 [10489.330321] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10489.333620] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10489.338184] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10489.342551] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10489.346644] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10489.350336] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10489.354162] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10489.358289] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10489.362894] workqueue events_long: flags=0x0 [10489.364367] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10489.369164] pending: lu_objects_flush [obdclass] [10489.372045] workqueue events_unbound: flags=0x2 [10489.374354] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10489.378207] pending: flush_memcg_stats_dwork, idle_cull_fn [10489.381674] workqueue events_power_efficient: flags=0x80 [10489.386055] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10489.390079] pending: neigh_periodic_work, do_cache_clean [10489.393946] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10489.398698] pending: neigh_periodic_work, check_lifetime [10489.401377] workqueue mm_percpu_wq: flags=0x8 [10489.402765] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10489.405860] pending: vmstat_update, lru_add_drain_per_cpu [10489.408979] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10489.412941] pending: vmstat_update [10489.414996] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10489.418833] pending: lru_add_drain_per_cpu BAR(48) [10489.421830] workqueue ipv6_addrconf: flags=0x40008 [10489.424352] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10489.427644] pending: addrconf_verify_work [10489.429554] workqueue knbd0-recv: flags=0x1a [10489.432145] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10489.435369] in-flight: 650:recv_work [10489.436982] workqueue ptlrpc_pinger: flags=0x2 [10489.439022] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10489.441277] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10489.444904] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7976s workers=4 idle: 36 466 29757 [10489.449389] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10479s workers=2 idle: 652 [10492.061619] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [10492.062508] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [10492.066079] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10492.066159] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10492.066165] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10492.066180] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10492.071523] Modules linked in: [10492.136018] [10492.136027] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10492.136045] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10492.136050] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10492.136054] RAX: 000000000000159a RBX: ffff9a61907912e0 RCX: 0000000000000001 [10492.136057] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10492.136059] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10492.136061] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10492.136063] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10492.136066] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10492.136069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10492.136071] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10492.136079] Call Trace: [10492.136085] [10492.136090] ? show_regs.cold.9+0x22/0x2f [10492.136098] ? watchdog_timer_fn.cold.6+0x62/0x10a [10492.136105] ? __hrtimer_run_queues+0x1c2/0x4d0 [10492.136109] ? watchdog+0x40/0x40 [10492.136113] ? hrtimer_interrupt+0x128/0x2c0 [10492.136117] ? smp_apic_timer_interrupt+0xa6/0x270 [10492.136126] ? apic_timer_interrupt+0xf/0x20 [10492.136131] [10492.136134] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10492.136139] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10492.136145] do_raw_spin_lock+0x177/0x190 [10492.136151] _raw_spin_lock+0x21/0x30 [10492.136155] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10492.141900] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10492.141978] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10492.141986] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10492.141989] RIP: 0010:native_safe_halt+0x2e/0x50 [10492.142003] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10492.142007] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10492.142012] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10492.142014] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10492.142016] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10492.142019] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10492.142021] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10492.142025] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10492.142027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10492.142029] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10492.142048] Call Trace: [10492.142054] [10492.142059] ? show_regs.cold.9+0x22/0x2f [10492.142069] ? watchdog_timer_fn.cold.6+0x62/0x10a [10492.142076] ? __hrtimer_run_queues+0x1c2/0x4d0 [10492.142080] ? watchdog+0x40/0x40 [10492.142084] ? hrtimer_interrupt+0x128/0x2c0 [10492.142088] ? smp_apic_timer_interrupt+0xa6/0x270 [10492.142094] ? apic_timer_interrupt+0xf/0x20 [10492.142098] [10492.142101] ? native_safe_halt+0x2e/0x50 [10492.142104] kvm_wait+0xcd/0xe0 [10492.142110] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10492.142116] do_raw_spin_lock+0x177/0x190 [10492.142120] _raw_spin_lock+0x21/0x30 [10492.142125] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10492.142648] ? do_raw_spin_unlock+0x75/0x190 [10492.142656] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10492.142849] ? do_wait_intr+0xf0/0xf0 [10492.142858] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10492.143068] mdc_reint+0x2d/0x110 [mdc] [10492.143110] mdc_unlink+0x658/0x870 [mdc] [10492.143139] lmv_unlink+0x34d/0xbd0 [lmv] [10492.150723] process_one_work+0x2c8/0x700 [10492.150745] worker_thread+0x296/0x6e0 [10492.150751] ? rescuer_thread+0x570/0x570 [10492.154767] ll_unlink+0x24d/0x7f0 [lustre] [10492.157560] kthread+0x1d1/0x200 [10492.157578] ? set_kthread_struct+0x70/0x70 [10492.157583] ret_from_fork+0x1f/0x30 [10492.486565] vfs_unlink+0x202/0x330 [10492.488033] do_unlinkat+0x2fa/0x460 [10492.492314] __x64_sys_unlink+0x37/0x50 [10492.495043] do_syscall_64+0xc1/0x440 [10492.496985] entry_SYSCALL_64_after_hwframe+0x49/0xae [10492.500953] RIP: 0033:0x7f83e5f5c10b [10492.503794] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10492.523671] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10492.528040] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10492.533348] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10492.552911] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10492.565413] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10492.577180] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10501.176028] rcu: INFO: rcu_sched self-detected stall on CPU [10501.180387] rcu: 0-....: (5349225 ticks this GP) idle=30a/1/0x4000000000000002 softirq=128519/128519 fqs=126395 [10501.186831] (t=8520227 jiffies g=262441 q=59741) [10501.189642] rcu: rcu_sched kthread starved for 7989417 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [10501.195977] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10501.202342] rcu: RCU grace-period kthread stack dump: [10501.205465] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10501.210223] Call Trace: [10501.211615] __schedule+0x351/0xcb0 [10501.213731] schedule+0xc0/0x180 [10501.215567] schedule_timeout+0xb4/0x190 [10501.217596] ? do_raw_spin_unlock+0x75/0x190 [10501.219657] ? __next_timer_interrupt+0x160/0x160 [10501.222263] ? prepare_to_swait_event+0xf3/0x140 [10501.226304] rcu_gp_kthread+0x8dd/0x15b0 [10501.228944] ? force_qs_rnp+0x2c0/0x2c0 [10501.231709] kthread+0x1d1/0x200 [10501.233103] ? set_kthread_struct+0x70/0x70 [10501.235059] ret_from_fork+0x1f/0x30 [10501.237385] rcu: Stack dump where RCU GP kthread last ran: [10501.240062] Sending NMI from CPU 0 to CPUs 1: [10501.245255] NMI backtrace for cpu 1 [10501.245262] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10501.245264] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10501.245265] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10501.245267] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10501.245270] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 [10501.245272] RAX: 000000000000710f RBX: ffff9a61907912e0 RCX: 0000000000000001 [10501.245273] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10501.245275] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10501.245277] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10501.245278] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10501.245419] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10501.245421] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10501.245422] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10501.245423] Call Trace: [10501.245424] [10501.245425] ? show_regs.cold.9+0x22/0x2f [10501.245426] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10501.245427] ? nmi_cpu_backtrace_handler+0x15/0x40 [10501.245428] ? nmi_handle+0xbd/0x210 [10501.245429] ? default_do_nmi+0x4b/0x1b0 [10501.245430] ? do_nmi+0x280/0x380 [10501.245431] ? end_repeat_nmi+0x16/0x1f [10501.245432] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10501.245433] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10501.245435] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10501.245436] [10501.245437] do_raw_spin_lock+0x177/0x190 [10501.245438] _raw_spin_lock+0x21/0x30 [10501.245439] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10501.245441] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10501.245442] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10501.245444] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10501.245445] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10501.245446] ? woken_wake_function+0x30/0x30 [10501.245447] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10501.245448] kthread+0x1d1/0x200 [10501.245449] ? set_kthread_struct+0x70/0x70 [10501.245450] ret_from_fork+0x1f/0x30 [10501.246033] NMI backtrace for cpu 0 [10501.360925] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10501.365843] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10501.369992] Call Trace: [10501.371020] [10501.372242] ? dump_stack+0xbb/0x10e [10501.374080] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10501.376277] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10501.378946] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10501.381326] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10501.383572] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10501.385301] ? rcu_dump_cpu_stacks+0x11b/0x171 [10501.387673] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10501.389883] ? update_process_times+0x6d/0xc0 [10501.392006] ? tick_sched_handle.isra.18+0x27/0x90 [10501.394404] ? tick_sched_timer+0x53/0xb0 [10501.396301] ? __hrtimer_run_queues+0x1c2/0x4d0 [10501.397647] ? tick_sched_handle.isra.18+0x90/0x90 [10501.399400] ? hrtimer_interrupt+0x128/0x2c0 [10501.400972] ? smp_apic_timer_interrupt+0xa6/0x270 [10501.403378] ? apic_timer_interrupt+0xf/0x20 [10501.405418] [10501.406728] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10501.409601] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10501.413074] ? do_raw_spin_lock+0x177/0x190 [10501.415220] ? _raw_spin_lock+0x21/0x30 [10501.417130] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10501.420039] ? obd_mod_free+0x46/0x140 [ptlrpc] [10501.422561] ? mdc_commit_open+0x91/0xb0 [mdc] [10501.424961] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10501.427477] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10501.431046] ? after_reply+0xe2c/0x1780 [ptlrpc] [10501.433990] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10501.436890] ? __next_timer_interrupt+0x160/0x160 [10501.439045] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10501.441384] ? do_wait_intr+0xf0/0xf0 [10501.443207] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10501.446510] ? kthread+0x1d1/0x200 [10501.448396] ? set_kthread_struct+0x70/0x70 [10501.450295] ? ret_from_fork+0x1f/0x30 [10519.522896] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8006s! [10519.526865] Showing busy workqueues and worker pools: [10519.533800] workqueue events: flags=0x0 [10519.537760] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10519.541292] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10519.548975] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10519.558879] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10519.562426] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10519.569039] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10519.577715] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10519.588236] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10519.593524] workqueue events_long: flags=0x0 [10519.600466] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10519.608509] pending: lu_objects_flush [obdclass] [10519.610218] workqueue events_unbound: flags=0x2 [10519.616044] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10519.621657] pending: flush_memcg_stats_dwork, idle_cull_fn [10519.627746] workqueue events_power_efficient: flags=0x80 [10519.633074] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10519.640929] pending: neigh_periodic_work, do_cache_clean [10519.647747] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10519.651074] pending: neigh_periodic_work, check_lifetime [10519.654214] workqueue mm_percpu_wq: flags=0x8 [10519.655769] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10519.659730] pending: vmstat_update, lru_add_drain_per_cpu [10519.664286] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10519.672041] pending: vmstat_update [10519.678758] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10519.686141] pending: lru_add_drain_per_cpu BAR(48) [10519.687827] workqueue ipv6_addrconf: flags=0x40008 [10519.697011] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10519.706771] pending: addrconf_verify_work [10519.711687] workqueue knbd0-recv: flags=0x1a [10519.718092] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10519.724784] in-flight: 650:recv_work [10519.727084] workqueue ptlrpc_pinger: flags=0x2 [10519.731045] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10519.734439] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10519.741286] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8006s workers=4 idle: 36 466 29757 [10519.752878] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10509s workers=2 idle: 652 [10520.060008] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [10520.062385] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10520.066257] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [10520.078618] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10520.078634] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10520.078717] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10520.079046] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10520.079058] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10520.079063] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10520.079067] RAX: 00000000000012c6 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10520.079070] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10520.079072] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10520.079074] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10520.079077] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10520.079080] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10520.079083] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10520.079085] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10520.079092] Call Trace: [10520.079258] [10520.079263] ? show_regs.cold.9+0x22/0x2f [10520.079272] ? watchdog_timer_fn.cold.6+0x62/0x10a [10520.079277] ? __hrtimer_run_queues+0x1c2/0x4d0 [10520.079281] ? watchdog+0x40/0x40 [10520.079284] ? hrtimer_interrupt+0x128/0x2c0 [10520.079288] ? smp_apic_timer_interrupt+0xa6/0x270 [10520.079295] ? apic_timer_interrupt+0xf/0x20 [10520.079299] [10520.079312] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10520.079318] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10520.079324] do_raw_spin_lock+0x177/0x190 [10520.079333] _raw_spin_lock+0x21/0x30 [10520.079338] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10520.079556] process_one_work+0x2c8/0x700 [10520.079564] worker_thread+0x296/0x6e0 [10520.079568] ? rescuer_thread+0x570/0x570 [10520.079572] kthread+0x1d1/0x200 [10520.079576] ? set_kthread_struct+0x70/0x70 [10520.079579] ret_from_fork+0x1f/0x30 [10520.167434] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10520.185262] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10520.189400] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10520.192978] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10520.194581] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10520.202743] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10520.206348] RAX: 0000000000003419 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10520.209557] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10520.212406] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10520.215858] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10520.219075] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10520.222446] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10520.226551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10520.229083] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10520.232233] Call Trace: [10520.233306] [10520.234179] ? show_regs.cold.9+0x22/0x2f [10520.236042] ? watchdog_timer_fn.cold.6+0x62/0x10a [10520.238088] ? __hrtimer_run_queues+0x1c2/0x4d0 [10520.240499] ? watchdog+0x40/0x40 [10520.241869] ? hrtimer_interrupt+0x128/0x2c0 [10520.243992] ? smp_apic_timer_interrupt+0xa6/0x270 [10520.245641] ? apic_timer_interrupt+0xf/0x20 [10520.247358] [10520.248374] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10520.250320] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10520.252564] do_raw_spin_lock+0x177/0x190 [10520.254374] _raw_spin_lock+0x21/0x30 [10520.255862] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10520.258294] ? do_raw_spin_unlock+0x75/0x190 [10520.260086] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10520.262537] ? do_wait_intr+0xf0/0xf0 [10520.264210] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10520.266329] mdc_reint+0x2d/0x110 [mdc] [10520.268282] mdc_unlink+0x658/0x870 [mdc] [10520.269919] lmv_unlink+0x34d/0xbd0 [lmv] [10520.271711] ll_unlink+0x24d/0x7f0 [lustre] [10520.273532] vfs_unlink+0x202/0x330 [10520.275290] do_unlinkat+0x2fa/0x460 [10520.276766] __x64_sys_unlink+0x37/0x50 [10520.278386] do_syscall_64+0xc1/0x440 [10520.279820] entry_SYSCALL_64_after_hwframe+0x49/0xae [10520.281690] RIP: 0033:0x7f83e5f5c10b [10520.283176] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10520.291068] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10520.294011] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10520.296914] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10520.299706] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10520.302108] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10520.304873] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10528.059098] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [10528.071426] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10528.122580] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10528.142582] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10528.151021] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10528.155382] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10528.179411] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10528.185098] RAX: 0000000000006808 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10528.188710] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10528.196601] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10528.206640] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10528.216532] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10528.234566] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10528.245614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10528.251776] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10528.263986] Call Trace: [10528.264809] [10528.274817] ? show_regs.cold.9+0x22/0x2f [10528.280657] ? watchdog_timer_fn.cold.6+0x62/0x10a [10528.290169] ? __hrtimer_run_queues+0x1c2/0x4d0 [10528.292512] ? watchdog+0x40/0x40 [10528.295457] ? hrtimer_interrupt+0x128/0x2c0 [10528.297438] ? smp_apic_timer_interrupt+0xa6/0x270 [10528.299208] ? apic_timer_interrupt+0xf/0x20 [10528.307296] [10528.309501] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10528.314418] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10528.317459] do_raw_spin_lock+0x177/0x190 [10528.319775] _raw_spin_lock+0x21/0x30 [10528.321669] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10528.326414] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10528.330493] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10528.338199] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10528.341617] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10528.347700] ? woken_wake_function+0x30/0x30 [10528.348733] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10528.352860] kthread+0x1d1/0x200 [10528.356485] ? set_kthread_struct+0x70/0x70 [10528.358669] ret_from_fork+0x1f/0x30 [10544.056088] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [10544.059711] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10544.099695] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10544.108035] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10544.112871] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10544.123232] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10544.133866] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10544.137389] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10544.140402] RDX: 00000000000004e1 RSI: 0000000000000003 RDI: ffff9a61907912e0 [10544.143555] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10544.148147] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10544.151577] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10544.154565] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10544.159934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10544.162148] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10544.166429] Call Trace: [10544.167850] [10544.168919] ? show_regs.cold.9+0x22/0x2f [10544.171441] ? watchdog_timer_fn.cold.6+0x62/0x10a [10544.173626] ? __hrtimer_run_queues+0x1c2/0x4d0 [10544.175725] ? watchdog+0x40/0x40 [10544.177285] ? hrtimer_interrupt+0x128/0x2c0 [10544.179093] ? smp_apic_timer_interrupt+0xa6/0x270 [10544.181540] ? apic_timer_interrupt+0xf/0x20 [10544.183618] [10544.185370] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10544.194304] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10544.196660] do_raw_spin_lock+0x177/0x190 [10544.199800] _raw_spin_lock+0x21/0x30 [10544.204804] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10544.208774] obd_mod_free+0x46/0x140 [ptlrpc] [10544.213942] mdc_commit_open+0x91/0xb0 [mdc] [10544.215540] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10544.222598] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10544.226583] after_reply+0xe2c/0x1780 [ptlrpc] [10544.231472] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10544.234883] ? __next_timer_interrupt+0x160/0x160 [10544.237536] ptlrpcd+0xa82/0xe40 [ptlrpc] [10544.239986] ? do_wait_intr+0xf0/0xf0 [10544.242740] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10544.245758] kthread+0x1d1/0x200 [10544.247526] ? set_kthread_struct+0x70/0x70 [10544.249400] ret_from_fork+0x1f/0x30 [10548.061678] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [10548.062294] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [10548.066057] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10548.066136] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10548.066144] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10548.066150] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10548.066924] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10548.066936] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10548.066940] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10548.066944] RAX: 000000000000528b RBX: ffff9a61907912e0 RCX: 0000000000000001 [10548.066947] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10548.066949] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10548.066951] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10548.066953] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10548.066957] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10548.066960] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10548.066963] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10548.066972] Call Trace: [10548.066976] [10548.066981] ? show_regs.cold.9+0x22/0x2f [10548.066992] ? watchdog_timer_fn.cold.6+0x62/0x10a [10548.066999] ? __hrtimer_run_queues+0x1c2/0x4d0 [10548.067003] ? watchdog+0x40/0x40 [10548.067008] ? hrtimer_interrupt+0x128/0x2c0 [10548.067012] ? smp_apic_timer_interrupt+0xa6/0x270 [10548.067020] ? apic_timer_interrupt+0xf/0x20 [10548.067025] [10548.067028] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10548.067032] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10548.067037] do_raw_spin_lock+0x177/0x190 [10548.067043] _raw_spin_lock+0x21/0x30 [10548.067048] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10548.067283] process_one_work+0x2c8/0x700 [10548.067293] worker_thread+0x296/0x6e0 [10548.067297] ? rescuer_thread+0x570/0x570 [10548.067301] kthread+0x1d1/0x200 [10548.067306] ? set_kthread_struct+0x70/0x70 [10548.067311] ret_from_fork+0x1f/0x30 [10548.184434] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10548.202106] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10548.206336] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10548.209796] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10548.212217] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10548.219117] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10548.221643] RAX: 000000000000689c RBX: ffff9a61907912e0 RCX: 0000000000000001 [10548.224412] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10548.227057] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10548.229592] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10548.234544] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10548.239039] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10548.252589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10548.254413] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10548.267222] Call Trace: [10548.270003] [10548.271100] ? show_regs.cold.9+0x22/0x2f [10548.274678] ? watchdog_timer_fn.cold.6+0x62/0x10a [10548.280182] ? __hrtimer_run_queues+0x1c2/0x4d0 [10548.285331] ? watchdog+0x40/0x40 [10548.290031] ? hrtimer_interrupt+0x128/0x2c0 [10548.291422] ? smp_apic_timer_interrupt+0xa6/0x270 [10548.297606] ? apic_timer_interrupt+0xf/0x20 [10548.300334] [10548.302052] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10548.305270] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10548.309297] do_raw_spin_lock+0x177/0x190 [10548.315251] _raw_spin_lock+0x21/0x30 [10548.319193] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10548.324187] ? do_raw_spin_unlock+0x75/0x190 [10548.328693] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10548.331456] ? do_wait_intr+0xf0/0xf0 [10548.333728] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10548.336235] mdc_reint+0x2d/0x110 [mdc] [10548.337361] mdc_unlink+0x658/0x870 [mdc] [10548.347109] lmv_unlink+0x34d/0xbd0 [lmv] [10548.351216] ll_unlink+0x24d/0x7f0 [lustre] [10548.356949] vfs_unlink+0x202/0x330 [10548.362187] do_unlinkat+0x2fa/0x460 [10548.365157] __x64_sys_unlink+0x37/0x50 [10548.369201] do_syscall_64+0xc1/0x440 [10548.372429] entry_SYSCALL_64_after_hwframe+0x49/0xae [10548.376797] RIP: 0033:0x7f83e5f5c10b [10548.379289] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10548.397135] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10548.405344] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10548.411176] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10548.417664] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10548.423610] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10548.435529] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10550.243157] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8037s! [10550.247664] Showing busy workqueues and worker pools: [10550.257166] workqueue events: flags=0x0 [10550.260080] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10550.269830] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10550.283091] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10550.289118] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10550.293678] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10550.298358] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10550.303480] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10550.308211] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10550.314529] workqueue events_long: flags=0x0 [10550.317068] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10550.325947] pending: lu_objects_flush [obdclass] [10550.334559] workqueue events_unbound: flags=0x2 [10550.339192] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10550.344007] pending: flush_memcg_stats_dwork, idle_cull_fn [10550.347661] workqueue events_power_efficient: flags=0x80 [10550.350202] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10550.354068] pending: neigh_periodic_work, do_cache_clean [10550.356750] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10550.360516] pending: neigh_periodic_work, check_lifetime [10550.363508] workqueue mm_percpu_wq: flags=0x8 [10550.365637] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10550.369667] pending: vmstat_update, lru_add_drain_per_cpu [10550.372406] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10550.376349] pending: vmstat_update [10550.378039] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10550.380971] pending: lru_add_drain_per_cpu BAR(48) [10550.383110] workqueue ipv6_addrconf: flags=0x40008 [10550.385303] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10550.387946] pending: addrconf_verify_work [10550.390123] workqueue knbd0-recv: flags=0x1a [10550.391981] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10550.394763] in-flight: 650:recv_work [10550.397176] workqueue ptlrpc_pinger: flags=0x2 [10550.399241] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10550.402620] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10550.405112] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8037s workers=4 idle: 36 466 29757 [10550.408426] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10540s workers=2 idle: 652 [10556.059028] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [10556.062270] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10556.085122] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10556.091353] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10556.095980] RIP: 0010:native_safe_halt+0x2e/0x50 [10556.098538] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10556.111397] RSP: 0018:ffffbc1d01cb7d48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10556.116494] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10556.120181] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10556.123990] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10556.127815] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10556.131449] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10556.134088] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10556.138202] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10556.141194] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10556.144738] Call Trace: [10556.147115] [10556.147862] ? show_regs.cold.9+0x22/0x2f [10556.150077] ? watchdog_timer_fn.cold.6+0x62/0x10a [10556.152628] ? __hrtimer_run_queues+0x1c2/0x4d0 [10556.155059] ? watchdog+0x40/0x40 [10556.156971] ? hrtimer_interrupt+0x128/0x2c0 [10556.159103] ? smp_apic_timer_interrupt+0xa6/0x270 [10556.161362] ? apic_timer_interrupt+0xf/0x20 [10556.163886] [10556.164920] ? native_safe_halt+0x2e/0x50 [10556.167268] kvm_wait+0xcd/0xe0 [10556.168662] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10556.170337] do_raw_spin_lock+0x177/0x190 [10556.172304] _raw_spin_lock+0x21/0x30 [10556.174191] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10556.177845] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10556.180562] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10556.184497] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10556.188654] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10556.192134] ? woken_wake_function+0x30/0x30 [10556.195200] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10556.198313] kthread+0x1d1/0x200 [10556.199895] ? set_kthread_struct+0x70/0x70 [10556.201720] ret_from_fork+0x1f/0x30 [10576.057876] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [10576.060494] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [10576.062023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [10576.062033] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10576.062490] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10576.062496] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10576.062498] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10576.062512] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10576.062516] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10576.062521] RAX: 000000000000288c RBX: ffff9a61907912e0 RCX: 0000000000000001 [10576.062523] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10576.062526] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10576.062528] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10576.062530] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10576.062533] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10576.062536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10576.062539] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10576.062546] Call Trace: [10576.062551] [10576.062556] ? show_regs.cold.9+0x22/0x2f [10576.062566] ? watchdog_timer_fn.cold.6+0x62/0x10a [10576.062572] ? __hrtimer_run_queues+0x1c2/0x4d0 [10576.062577] ? watchdog+0x40/0x40 [10576.062581] ? hrtimer_interrupt+0x128/0x2c0 [10576.062586] ? smp_apic_timer_interrupt+0xa6/0x270 [10576.062594] ? apic_timer_interrupt+0xf/0x20 [10576.062598] [10576.062601] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10576.062605] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10576.062611] do_raw_spin_lock+0x177/0x190 [10576.062615] _raw_spin_lock+0x21/0x30 [10576.062621] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10576.062881] ? do_raw_spin_unlock+0x75/0x190 [10576.062887] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10576.063071] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) [10576.063099] ? do_wait_intr+0xf0/0xf0 [10576.063104] obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10576.063141] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10576.063145] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10576.063147] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10576.063155] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10576.063158] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10576.063162] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10576.063164] RDX: 0000000000000e54 RSI: 0000000000000003 RDI: ffff9a61907912e0 [10576.063166] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10576.063168] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10576.063171] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10576.063173] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10576.063176] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10576.063178] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10576.063185] Call Trace: [10576.063188] [10576.063191] ? show_regs.cold.9+0x22/0x2f [10576.063197] ? watchdog_timer_fn.cold.6+0x62/0x10a [10576.063203] ? __hrtimer_run_queues+0x1c2/0x4d0 [10576.063207] ? watchdog+0x40/0x40 [10576.063213] ? hrtimer_interrupt+0x128/0x2c0 [10576.063217] ? smp_apic_timer_interrupt+0xa6/0x270 [10576.063225] ? apic_timer_interrupt+0xf/0x20 [10576.063229] [10576.063233] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10576.063238] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10576.063242] do_raw_spin_lock+0x177/0x190 [10576.063246] _raw_spin_lock+0x21/0x30 [10576.063111] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10576.063251] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10576.063618] mdc_reint+0x2d/0x110 [mdc] [10576.063654] mdc_unlink+0x658/0x870 [mdc] [10576.063680] lmv_unlink+0x34d/0xbd0 [lmv] [10576.063710] ll_unlink+0x24d/0x7f0 [lustre] [10576.063822] vfs_unlink+0x202/0x330 [10576.063829] do_unlinkat+0x2fa/0x460 [10576.063834] __x64_sys_unlink+0x37/0x50 [10576.063837] do_syscall_64+0xc1/0x440 [10576.063844] entry_SYSCALL_64_after_hwframe+0x49/0xae [10576.063849] RIP: 0033:0x7f83e5f5c10b [10576.063855] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10576.063858] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10576.063862] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10576.063864] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10576.063866] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10576.063868] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10576.063870] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10576.066430] obd_mod_free+0x46/0x140 [ptlrpc] [10576.067498] mdc_commit_open+0x91/0xb0 [mdc] [10576.070635] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10576.070900] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10576.071091] after_reply+0xe2c/0x1780 [ptlrpc] [10576.071281] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10576.071526] ? __next_timer_interrupt+0x160/0x160 [10576.071537] ptlrpcd+0xa82/0xe40 [ptlrpc] [10576.071737] ? do_wait_intr+0xf0/0xf0 [10576.071747] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10576.071945] kthread+0x1d1/0x200 [10576.071951] ? set_kthread_struct+0x70/0x70 [10576.071956] ret_from_fork+0x1f/0x30 [10576.498574] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10576.522495] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10576.530708] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10576.536947] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10576.540290] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10576.544752] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10576.553134] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10576.557771] RAX: 0000000000007d3c RBX: ffff9a61907912e0 RCX: 0000000000000001 [10576.563650] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10576.571093] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10576.574141] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10576.579070] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10576.582477] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10576.586063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10576.590687] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10576.594938] Call Trace: [10576.596031] [10576.597110] ? show_regs.cold.9+0x22/0x2f [10576.598986] ? watchdog_timer_fn.cold.6+0x62/0x10a [10576.601422] ? __hrtimer_run_queues+0x1c2/0x4d0 [10576.603138] ? watchdog+0x40/0x40 [10576.605007] ? hrtimer_interrupt+0x128/0x2c0 [10576.607888] ? smp_apic_timer_interrupt+0xa6/0x270 [10576.609609] ? apic_timer_interrupt+0xf/0x20 [10576.612236] [10576.613450] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10576.619420] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10576.622952] do_raw_spin_lock+0x177/0x190 [10576.627753] _raw_spin_lock+0x21/0x30 [10576.629219] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10576.637114] process_one_work+0x2c8/0x700 [10576.639072] worker_thread+0x296/0x6e0 [10576.641086] ? rescuer_thread+0x570/0x570 [10576.645429] kthread+0x1d1/0x200 [10576.650387] ? set_kthread_struct+0x70/0x70 [10576.653902] ret_from_fork+0x1f/0x30 [10580.456522] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8067s! [10580.468749] Showing busy workqueues and worker pools: [10580.471087] workqueue events: flags=0x0 [10580.473749] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10580.478709] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10580.489436] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10580.496812] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10580.506145] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10580.513950] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10580.522439] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10580.539498] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10580.551924] workqueue events_long: flags=0x0 [10580.562163] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10580.570987] pending: lu_objects_flush [obdclass] [10580.574186] workqueue events_unbound: flags=0x2 [10580.579569] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10580.586733] pending: flush_memcg_stats_dwork, idle_cull_fn [10580.595838] workqueue events_power_efficient: flags=0x80 [10580.599067] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10580.605276] pending: neigh_periodic_work, do_cache_clean [10580.608164] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10580.614740] pending: neigh_periodic_work, check_lifetime [10580.617345] workqueue mm_percpu_wq: flags=0x8 [10580.619360] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10580.621990] pending: vmstat_update, lru_add_drain_per_cpu [10580.623854] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10580.631289] pending: vmstat_update [10580.636038] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10580.639040] pending: lru_add_drain_per_cpu BAR(48) [10580.643042] workqueue ipv6_addrconf: flags=0x40008 [10580.650353] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10580.657697] pending: addrconf_verify_work [10580.661612] workqueue knbd0-recv: flags=0x1a [10580.664019] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10580.666252] in-flight: 650:recv_work [10580.669232] workqueue ptlrpc_pinger: flags=0x2 [10580.670810] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10580.675191] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10580.689305] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8067s workers=4 idle: 36 466 29757 [10580.700243] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10570s workers=2 idle: 652 [10584.063676] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [10584.071800] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10584.131847] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10584.148838] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10584.155381] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10584.162082] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10584.187303] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10584.197111] RAX: 0000000000006a63 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10584.205606] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10584.219179] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10584.228296] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10584.236155] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10584.245011] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10584.255924] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10584.258111] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10584.268396] Call Trace: [10584.270331] [10584.272019] ? show_regs.cold.9+0x22/0x2f [10584.278009] ? watchdog_timer_fn.cold.6+0x62/0x10a [10584.285707] ? __hrtimer_run_queues+0x1c2/0x4d0 [10584.289838] ? watchdog+0x40/0x40 [10584.291900] ? hrtimer_interrupt+0x128/0x2c0 [10584.293210] ? smp_apic_timer_interrupt+0xa6/0x270 [10584.304305] ? apic_timer_interrupt+0xf/0x20 [10584.307865] [10584.318809] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10584.323586] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10584.329889] do_raw_spin_lock+0x177/0x190 [10584.332039] _raw_spin_lock+0x21/0x30 [10584.336237] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10584.342070] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10584.346164] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10584.351390] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10584.355008] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10584.361567] ? woken_wake_function+0x30/0x30 [10584.364818] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10584.371843] kthread+0x1d1/0x200 [10584.377223] ? set_kthread_struct+0x70/0x70 [10584.380851] ret_from_fork+0x1f/0x30 [10604.061942] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [10604.063440] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [10604.072605] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10604.072689] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10604.072696] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10604.072702] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10604.073008] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10604.073019] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10604.073023] RSP: 0018:ffffbc1d00c97dd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10604.073027] RAX: 0000000000001a09 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10604.073030] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10604.073032] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10604.073034] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10604.073036] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10604.073040] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10604.073043] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10604.073045] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10604.073053] Call Trace: [10604.073058] [10604.073064] ? show_regs.cold.9+0x22/0x2f [10604.073073] ? watchdog_timer_fn.cold.6+0x62/0x10a [10604.073080] ? __hrtimer_run_queues+0x1c2/0x4d0 [10604.073084] ? watchdog+0x40/0x40 [10604.073099] ? hrtimer_interrupt+0x128/0x2c0 [10604.073103] ? smp_apic_timer_interrupt+0xa6/0x270 [10604.073113] ? apic_timer_interrupt+0xf/0x20 [10604.073118] [10604.073120] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10604.073125] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10604.073130] do_raw_spin_lock+0x177/0x190 [10604.073135] _raw_spin_lock+0x21/0x30 [10604.073139] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10604.075236] process_one_work+0x2c8/0x700 [10604.075247] worker_thread+0x296/0x6e0 [10604.075251] ? rescuer_thread+0x570/0x570 [10604.075255] kthread+0x1d1/0x200 [10604.075259] ? set_kthread_struct+0x70/0x70 [10604.075263] ret_from_fork+0x1f/0x30 [10604.346597] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10604.384554] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10604.390621] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10604.395973] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10604.398450] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10604.406616] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10604.410078] RAX: 0000000000001409 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10604.413155] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10604.417045] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10604.431370] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10604.442111] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10604.454206] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10604.462020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10604.472672] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10604.479250] Call Trace: [10604.480645] [10604.481769] ? show_regs.cold.9+0x22/0x2f [10604.490826] ? watchdog_timer_fn.cold.6+0x62/0x10a [10604.497228] ? __hrtimer_run_queues+0x1c2/0x4d0 [10604.499600] ? watchdog+0x40/0x40 [10604.500917] ? hrtimer_interrupt+0x128/0x2c0 [10604.507578] ? smp_apic_timer_interrupt+0xa6/0x270 [10604.510180] ? apic_timer_interrupt+0xf/0x20 [10604.516213] [10604.517022] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10604.522704] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10604.528761] do_raw_spin_lock+0x177/0x190 [10604.529947] _raw_spin_lock+0x21/0x30 [10604.531709] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10604.536770] ? do_raw_spin_unlock+0x75/0x190 [10604.543302] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10604.547629] ? do_wait_intr+0xf0/0xf0 [10604.549546] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10604.558261] mdc_reint+0x2d/0x110 [mdc] [10604.559752] mdc_unlink+0x658/0x870 [mdc] [10604.565119] lmv_unlink+0x34d/0xbd0 [lmv] [10604.575751] ll_unlink+0x24d/0x7f0 [lustre] [10604.579214] vfs_unlink+0x202/0x330 [10604.583654] do_unlinkat+0x2fa/0x460 [10604.584631] __x64_sys_unlink+0x37/0x50 [10604.585560] do_syscall_64+0xc1/0x440 [10604.586409] entry_SYSCALL_64_after_hwframe+0x49/0xae [10604.592607] RIP: 0033:0x7f83e5f5c10b [10604.594630] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10604.604707] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10604.608326] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10604.619632] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10604.628273] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10604.635578] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10604.646200] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10608.056020] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [10608.060789] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10608.092956] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10608.102624] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10608.104623] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10608.108792] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10608.121210] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10608.124774] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10608.128980] RDX: 00000000000052a6 RSI: 0000000000000003 RDI: ffff9a61907912e0 [10608.131929] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10608.135131] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10608.137921] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10608.141466] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10608.147079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10608.149123] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10608.158186] Call Trace: [10608.160456] [10608.161831] ? show_regs.cold.9+0x22/0x2f [10608.163475] ? watchdog_timer_fn.cold.6+0x62/0x10a [10608.165443] ? __hrtimer_run_queues+0x1c2/0x4d0 [10608.167344] ? watchdog+0x40/0x40 [10608.168755] ? hrtimer_interrupt+0x128/0x2c0 [10608.170801] ? smp_apic_timer_interrupt+0xa6/0x270 [10608.175528] ? apic_timer_interrupt+0xf/0x20 [10608.177355] [10608.178457] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10608.182903] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10608.187214] do_raw_spin_lock+0x177/0x190 [10608.190032] _raw_spin_lock+0x21/0x30 [10608.193422] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10608.200002] obd_mod_free+0x46/0x140 [ptlrpc] [10608.205736] mdc_commit_open+0x91/0xb0 [mdc] [10608.207862] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10608.213393] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10608.218410] after_reply+0xe2c/0x1780 [ptlrpc] [10608.223964] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10608.229484] ? __next_timer_interrupt+0x160/0x160 [10608.231255] ptlrpcd+0xa82/0xe40 [ptlrpc] [10608.233084] ? do_wait_intr+0xf0/0xf0 [10608.234558] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10608.236833] kthread+0x1d1/0x200 [10608.238040] ? set_kthread_struct+0x70/0x70 [10608.239637] ret_from_fork+0x1f/0x30 [10611.172315] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8097s! [10611.177828] Showing busy workqueues and worker pools: [10611.182399] workqueue events: flags=0x0 [10611.183767] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10611.186605] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10611.191564] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10611.194898] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10611.197912] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10611.200518] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10611.205477] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10611.213345] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10611.216308] workqueue events_long: flags=0x0 [10611.218213] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10611.220497] pending: lu_objects_flush [obdclass] [10611.221843] workqueue events_unbound: flags=0x2 [10611.223474] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10611.224959] pending: flush_memcg_stats_dwork, idle_cull_fn [10611.227795] workqueue events_power_efficient: flags=0x80 [10611.229338] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10611.231863] pending: neigh_periodic_work, do_cache_clean [10611.233585] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10611.236712] pending: neigh_periodic_work, check_lifetime [10611.238656] workqueue mm_percpu_wq: flags=0x8 [10611.240496] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10611.242789] pending: vmstat_update, lru_add_drain_per_cpu [10611.244667] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10611.247444] pending: vmstat_update [10611.248637] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10611.251341] pending: lru_add_drain_per_cpu BAR(48) [10611.253407] workqueue ipv6_addrconf: flags=0x40008 [10611.255459] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10611.263574] pending: addrconf_verify_work [10611.265695] workqueue knbd0-recv: flags=0x1a [10611.267425] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10611.269709] in-flight: 650:recv_work [10611.271471] workqueue ptlrpc_pinger: flags=0x2 [10611.273527] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10611.276260] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10611.279304] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8098s workers=4 idle: 36 466 29757 [10611.282362] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10601s workers=2 idle: 652 [10612.059516] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [10612.064564] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10612.089673] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10612.096549] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10612.101287] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10612.105222] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10612.115023] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10612.119031] RAX: 0000000000003e9a RBX: ffff9a61907912e0 RCX: 0000000000000001 [10612.122278] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10612.126985] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10612.130166] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10612.136979] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10612.144976] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10612.149072] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10612.151865] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10612.155438] Call Trace: [10612.156200] [10612.157275] ? show_regs.cold.9+0x22/0x2f [10612.159102] ? watchdog_timer_fn.cold.6+0x62/0x10a [10612.161630] ? __hrtimer_run_queues+0x1c2/0x4d0 [10612.163699] ? watchdog+0x40/0x40 [10612.165127] ? hrtimer_interrupt+0x128/0x2c0 [10612.166715] ? smp_apic_timer_interrupt+0xa6/0x270 [10612.168701] ? apic_timer_interrupt+0xf/0x20 [10612.170962] [10612.171912] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10612.174043] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10612.176703] do_raw_spin_lock+0x177/0x190 [10612.178639] _raw_spin_lock+0x21/0x30 [10612.180356] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10612.183536] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10612.186494] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10612.190039] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10612.193041] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10612.196987] ? woken_wake_function+0x30/0x30 [10612.200627] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10612.205040] kthread+0x1d1/0x200 [10612.208716] ? set_kthread_struct+0x70/0x70 [10612.211504] ret_from_fork+0x1f/0x30 [10632.060026] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [10632.062070] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [10632.062357] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10632.062442] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10632.062448] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10632.062456] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10632.067281] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10632.067366] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10632.067374] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10632.067377] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10632.067392] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10632.067397] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [10632.067402] RAX: 0000000000004ba1 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10632.067404] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10632.067406] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10632.067409] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10632.067411] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10632.067414] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10632.067417] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10632.067419] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10632.067426] Call Trace: [10632.067431] [10632.067436] ? show_regs.cold.9+0x22/0x2f [10632.067445] ? watchdog_timer_fn.cold.6+0x62/0x10a [10632.067458] ? __hrtimer_run_queues+0x1c2/0x4d0 [10632.067463] ? watchdog+0x40/0x40 [10632.067467] ? hrtimer_interrupt+0x128/0x2c0 [10632.067472] ? smp_apic_timer_interrupt+0xa6/0x270 [10632.067481] ? apic_timer_interrupt+0xf/0x20 [10632.067485] [10632.067489] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10632.067494] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10632.067498] do_raw_spin_lock+0x177/0x190 [10632.067503] _raw_spin_lock+0x21/0x30 [10632.067632] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10632.067928] ? do_raw_spin_unlock+0x75/0x190 [10632.067933] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10632.093077] [10632.093087] RIP: 0010:native_safe_halt+0x2e/0x50 [10632.093105] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10632.093110] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10632.093115] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10632.093118] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10632.093121] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10632.093124] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10632.093127] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10632.093132] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10632.093135] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10632.093138] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10632.093147] Call Trace: [10632.093154] [10632.093159] ? show_regs.cold.9+0x22/0x2f [10632.093169] ? watchdog_timer_fn.cold.6+0x62/0x10a [10632.093177] ? __hrtimer_run_queues+0x1c2/0x4d0 [10632.093181] ? watchdog+0x40/0x40 [10632.093186] ? hrtimer_interrupt+0x128/0x2c0 [10632.093190] ? smp_apic_timer_interrupt+0xa6/0x270 [10632.093197] ? apic_timer_interrupt+0xf/0x20 [10632.093203] [10632.093206] ? native_safe_halt+0x2e/0x50 [10632.093210] kvm_wait+0xcd/0xe0 [10632.093330] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10632.093343] do_raw_spin_lock+0x177/0x190 [10632.093347] _raw_spin_lock+0x21/0x30 [10632.093351] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10632.099657] ? do_wait_intr+0xf0/0xf0 [10632.099682] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10632.100111] mdc_reint+0x2d/0x110 [mdc] [10632.100148] mdc_unlink+0x658/0x870 [mdc] [10632.100178] lmv_unlink+0x34d/0xbd0 [lmv] [10632.100210] ll_unlink+0x24d/0x7f0 [lustre] [10632.100333] vfs_unlink+0x202/0x330 [10632.100341] do_unlinkat+0x2fa/0x460 [10632.100347] __x64_sys_unlink+0x37/0x50 [10632.100351] do_syscall_64+0xc1/0x440 [10632.100358] entry_SYSCALL_64_after_hwframe+0x49/0xae [10632.100367] RIP: 0033:0x7f83e5f5c10b [10632.100375] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10632.100379] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10632.100385] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10632.100387] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10632.100389] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10632.100392] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10632.100394] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10632.387843] process_one_work+0x2c8/0x700 [10632.389306] worker_thread+0x296/0x6e0 [10632.390802] ? rescuer_thread+0x570/0x570 [10632.392732] kthread+0x1d1/0x200 [10632.393721] ? set_kthread_struct+0x70/0x70 [10632.395584] ret_from_fork+0x1f/0x30 [10636.056580] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [10636.059861] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10636.076304] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10636.080428] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10636.083761] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10636.085986] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10636.092947] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10636.095855] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10636.098187] RDX: 0000000000004f5d RSI: 0000000000000003 RDI: ffff9a61907912e0 [10636.100964] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10636.103530] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10636.106441] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10636.108947] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10636.111834] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10636.113957] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10636.116645] Call Trace: [10636.117628] [10636.118438] ? show_regs.cold.9+0x22/0x2f [10636.119909] ? watchdog_timer_fn.cold.6+0x62/0x10a [10636.121649] ? __hrtimer_run_queues+0x1c2/0x4d0 [10636.123242] ? watchdog+0x40/0x40 [10636.124705] ? hrtimer_interrupt+0x128/0x2c0 [10636.126743] ? smp_apic_timer_interrupt+0xa6/0x270 [10636.128716] ? apic_timer_interrupt+0xf/0x20 [10636.130338] [10636.131150] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10636.133275] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10636.135394] do_raw_spin_lock+0x177/0x190 [10636.137262] _raw_spin_lock+0x21/0x30 [10636.138713] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10636.141140] obd_mod_free+0x46/0x140 [ptlrpc] [10636.142888] mdc_commit_open+0x91/0xb0 [mdc] [10636.144940] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10636.147452] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10636.149505] after_reply+0xe2c/0x1780 [ptlrpc] [10636.151495] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10636.153895] ? __next_timer_interrupt+0x160/0x160 [10636.155639] ptlrpcd+0xa82/0xe40 [ptlrpc] [10636.157650] ? do_wait_intr+0xf0/0xf0 [10636.158944] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10636.161052] kthread+0x1d1/0x200 [10636.162163] ? set_kthread_struct+0x70/0x70 [10636.163628] ret_from_fork+0x1f/0x30 [10640.062752] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ldlm_bl_01:15916] [10640.079790] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10640.140136] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10640.155339] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10640.163028] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10640.170260] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10640.181084] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10640.188671] RAX: 0000000000006611 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10640.196321] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10640.206291] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10640.214493] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10640.228069] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10640.230537] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10640.238364] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10640.244345] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10640.249376] Call Trace: [10640.250970] [10640.251758] ? show_regs.cold.9+0x22/0x2f [10640.254087] ? watchdog_timer_fn.cold.6+0x62/0x10a [10640.257051] ? __hrtimer_run_queues+0x1c2/0x4d0 [10640.263194] ? watchdog+0x40/0x40 [10640.265173] ? hrtimer_interrupt+0x128/0x2c0 [10640.267356] ? smp_apic_timer_interrupt+0xa6/0x270 [10640.274483] ? apic_timer_interrupt+0xf/0x20 [10640.277261] [10640.278424] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10640.280926] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10640.285588] do_raw_spin_lock+0x177/0x190 [10640.287651] _raw_spin_lock+0x21/0x30 [10640.289023] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10640.299361] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10640.307620] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10640.311405] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10640.317432] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10640.322482] ? woken_wake_function+0x30/0x30 [10640.323875] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10640.335398] kthread+0x1d1/0x200 [10640.336569] ? set_kthread_struct+0x70/0x70 [10640.344433] ret_from_fork+0x1f/0x30 [10641.377048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8128s! [10641.380173] Showing busy workqueues and worker pools: [10641.382577] workqueue events: flags=0x0 [10641.384496] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10641.388009] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10641.392027] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10641.395562] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10641.398508] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10641.401497] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10641.405095] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10641.408064] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10641.411901] workqueue events_long: flags=0x0 [10641.413855] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10641.417102] pending: lu_objects_flush [obdclass] [10641.419704] workqueue events_unbound: flags=0x2 [10641.421813] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10641.424756] pending: flush_memcg_stats_dwork, idle_cull_fn [10641.427552] workqueue events_power_efficient: flags=0x80 [10641.429557] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10641.432828] pending: neigh_periodic_work, do_cache_clean [10641.435552] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10641.438892] pending: neigh_periodic_work, check_lifetime [10641.441345] workqueue mm_percpu_wq: flags=0x8 [10641.443597] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10641.446762] pending: vmstat_update, lru_add_drain_per_cpu [10641.448820] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10641.451624] pending: vmstat_update [10641.453717] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10641.456339] pending: lru_add_drain_per_cpu BAR(48) [10641.458718] workqueue ipv6_addrconf: flags=0x40008 [10641.461334] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10641.463998] pending: addrconf_verify_work [10641.465872] workqueue knbd0-recv: flags=0x1a [10641.467740] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10641.470601] in-flight: 650:recv_work [10641.472608] workqueue ptlrpc_pinger: flags=0x2 [10641.474268] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10641.477277] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10641.480030] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8128s workers=4 idle: 36 466 29757 [10641.483484] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10631s workers=2 idle: 652 [10660.060342] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u8:0:8] [10660.062023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writemany:36352] [10660.062521] Modules linked in: [10660.064038] Modules linked in: [10660.064040] lustre(O) osp(O) [10660.064749] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10660.064818] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10660.064824] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10660.064827] RIP: 0010:native_safe_halt+0x2e/0x50 [10660.064840] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10660.064844] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10660.064849] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10660.064851] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10660.064853] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10660.064855] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10660.064858] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10660.064861] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10660.064864] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10660.064866] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10660.064873] Call Trace: [10660.064878] [10660.064882] ? show_regs.cold.9+0x22/0x2f [10660.064891] ? watchdog_timer_fn.cold.6+0x62/0x10a [10660.064897] ? __hrtimer_run_queues+0x1c2/0x4d0 [10660.064901] ? watchdog+0x40/0x40 [10660.064905] ? hrtimer_interrupt+0x128/0x2c0 [10660.064908] ? smp_apic_timer_interrupt+0xa6/0x270 [10660.064914] ? apic_timer_interrupt+0xf/0x20 [10660.064918] [10660.064921] ? native_safe_halt+0x2e/0x50 [10660.064924] kvm_wait+0xcd/0xe0 [10660.064931] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10660.064939] do_raw_spin_lock+0x177/0x190 [10660.064943] _raw_spin_lock+0x21/0x30 [10660.064948] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10660.066508] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10660.066575] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10660.066580] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10660.066586] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10660.067883] ? do_raw_spin_unlock+0x75/0x190 [10660.067896] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10660.068144] ? do_wait_intr+0xf0/0xf0 [10660.068152] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10660.068509] mdc_reint+0x2d/0x110 [mdc] [10660.068558] mdc_unlink+0x658/0x870 [mdc] [10660.068584] lmv_unlink+0x34d/0xbd0 [lmv] [10660.068610] ll_unlink+0x24d/0x7f0 [lustre] [10660.068740] vfs_unlink+0x202/0x330 [10660.068749] do_unlinkat+0x2fa/0x460 [10660.068754] __x64_sys_unlink+0x37/0x50 [10660.068757] do_syscall_64+0xc1/0x440 [10660.068763] entry_SYSCALL_64_after_hwframe+0x49/0xae [10660.068771] RIP: 0033:0x7f83e5f5c10b [10660.068778] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10660.068782] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10660.068787] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10660.068790] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10660.068792] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10660.068794] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10660.068797] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10660.182799] [10660.183415] RIP: 0010:native_safe_halt+0x2e/0x50 [10660.184843] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10660.191103] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10660.194345] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10660.196308] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10660.198832] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10660.201080] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10660.203320] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10660.205410] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10660.208169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10660.211293] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10660.213971] Call Trace: [10660.215020] [10660.215742] ? show_regs.cold.9+0x22/0x2f [10660.217688] ? watchdog_timer_fn.cold.6+0x62/0x10a [10660.219347] ? __hrtimer_run_queues+0x1c2/0x4d0 [10660.221233] ? watchdog+0x40/0x40 [10660.222613] ? hrtimer_interrupt+0x128/0x2c0 [10660.224124] ? smp_apic_timer_interrupt+0xa6/0x270 [10660.225646] ? apic_timer_interrupt+0xf/0x20 [10660.226956] [10660.227499] ? native_safe_halt+0x2e/0x50 [10660.228927] kvm_wait+0xcd/0xe0 [10660.230075] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10660.231839] do_raw_spin_lock+0x177/0x190 [10660.232974] _raw_spin_lock+0x21/0x30 [10660.234255] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10660.236446] process_one_work+0x2c8/0x700 [10660.237636] worker_thread+0x296/0x6e0 [10660.238751] ? rescuer_thread+0x570/0x570 [10660.240148] kthread+0x1d1/0x200 [10660.241259] ? set_kthread_struct+0x70/0x70 [10660.242486] ret_from_fork+0x1f/0x30 [10668.058572] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ptlrpcd_rcv:2348] [10668.061833] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [10668.065084] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10668.065171] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10668.065178] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10668.065180] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10668.065195] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10668.065200] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10668.065205] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10668.065208] RDX: 00000000000057a7 RSI: 0000000000000003 RDI: ffff9a61907912e0 [10668.065210] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10668.065212] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10668.065214] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10668.065218] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10668.065221] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10668.065224] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10668.065232] Call Trace: [10668.065238] [10668.065245] ? show_regs.cold.9+0x22/0x2f [10668.065255] ? watchdog_timer_fn.cold.6+0x62/0x10a [10668.065262] ? __hrtimer_run_queues+0x1c2/0x4d0 [10668.065266] ? watchdog+0x40/0x40 [10668.065270] ? hrtimer_interrupt+0x128/0x2c0 [10668.065274] ? smp_apic_timer_interrupt+0xa6/0x270 [10668.065299] ? apic_timer_interrupt+0xf/0x20 [10668.065303] [10668.065307] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10668.065312] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10668.065317] do_raw_spin_lock+0x177/0x190 [10668.065321] _raw_spin_lock+0x21/0x30 [10668.065327] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10668.073282] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10668.073359] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10668.073367] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10668.073517] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10668.073532] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10668.073536] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10668.073541] RAX: 0000000000003f83 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10668.073544] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10668.073546] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10668.073548] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10668.073551] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10668.073555] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10668.073558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10668.073567] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10668.073575] Call Trace: [10668.073582] [10668.073588] ? show_regs.cold.9+0x22/0x2f [10668.073599] ? watchdog_timer_fn.cold.6+0x62/0x10a [10668.073606] ? __hrtimer_run_queues+0x1c2/0x4d0 [10668.073610] ? watchdog+0x40/0x40 [10668.073614] ? hrtimer_interrupt+0x128/0x2c0 [10668.073618] ? smp_apic_timer_interrupt+0xa6/0x270 [10668.073627] ? apic_timer_interrupt+0xf/0x20 [10668.073632] [10668.073635] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10668.073640] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10668.073647] do_raw_spin_lock+0x177/0x190 [10668.073652] _raw_spin_lock+0x21/0x30 [10668.073658] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10668.074534] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10668.120915] obd_mod_free+0x46/0x140 [ptlrpc] [10668.121394] mdc_commit_open+0x91/0xb0 [mdc] [10668.121434] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10668.122416] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10668.123396] after_reply+0xe2c/0x1780 [ptlrpc] [10668.123695] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10668.138695] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10668.139251] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10668.139513] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10668.139847] ? woken_wake_function+0x30/0x30 [10668.139858] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10668.140158] kthread+0x1d1/0x200 [10668.140285] ? set_kthread_struct+0x70/0x70 [10668.140291] ret_from_fork+0x1f/0x30 [10668.455440] ? __next_timer_interrupt+0x160/0x160 [10668.459849] ptlrpcd+0xa82/0xe40 [ptlrpc] [10668.464562] ? do_wait_intr+0xf0/0xf0 [10668.465942] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10668.469228] kthread+0x1d1/0x200 [10668.470533] ? set_kthread_struct+0x70/0x70 [10668.475102] ret_from_fork+0x1f/0x30 [10671.584522] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8158s! [10671.593696] Showing busy workqueues and worker pools: [10671.596544] workqueue events: flags=0x0 [10671.598310] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10671.600923] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10671.604227] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10671.606715] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10671.608928] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10671.610963] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10671.616422] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10671.619901] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10671.626595] workqueue events_long: flags=0x0 [10671.628669] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10671.634102] pending: lu_objects_flush [obdclass] [10671.637306] workqueue events_unbound: flags=0x2 [10671.640818] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10671.644241] pending: flush_memcg_stats_dwork, idle_cull_fn [10671.647932] workqueue events_power_efficient: flags=0x80 [10671.651071] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10671.654494] pending: neigh_periodic_work, do_cache_clean [10671.657489] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10671.660783] pending: neigh_periodic_work, check_lifetime [10671.664889] workqueue mm_percpu_wq: flags=0x8 [10671.667079] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10671.669563] pending: vmstat_update, lru_add_drain_per_cpu [10671.672054] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10671.677863] pending: vmstat_update [10671.679899] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10671.683620] pending: lru_add_drain_per_cpu BAR(48) [10671.686535] workqueue ipv6_addrconf: flags=0x40008 [10671.690104] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10671.694056] pending: addrconf_verify_work [10671.696440] workqueue knbd0-recv: flags=0x1a [10671.698975] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10671.702093] in-flight: 650:recv_work [10671.704247] workqueue ptlrpc_pinger: flags=0x2 [10671.707520] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10671.711356] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10671.714546] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8158s workers=4 idle: 36 466 29757 [10671.718789] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10661s workers=2 idle: 652 [10681.180206] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [10681.183110] rcu: 0-....: (5471340 ticks this GP) idle=30a/1/0x4000000000000000 softirq=128519/128519 fqs=126395 [10681.194416] (detected by 1, t=8700221 jiffies, g=262441, q=59753) [10681.200182] Sending NMI from CPU 1 to CPUs 0: [10681.204662] NMI backtrace for cpu 0 [10681.204669] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10681.204676] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10681.204678] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10681.204680] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 27 56 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10681.204682] RSP: 0018:ffffbc1d019bfbc8 EFLAGS: 00000206 [10681.204685] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10681.204687] RDX: 0000000000001f29 RSI: 0000000000000003 RDI: ffff9a61907912e0 [10681.204688] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10681.204690] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10681.204691] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10681.204693] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10681.204695] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10681.204696] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10681.204698] Call Trace: [10681.204699] [10681.204700] ? show_regs.cold.9+0x22/0x2f [10681.204701] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10681.204703] ? nmi_cpu_backtrace_handler+0x15/0x40 [10681.204704] ? nmi_handle+0xbd/0x210 [10681.204705] ? default_do_nmi+0x4b/0x1b0 [10681.204706] ? do_nmi+0x280/0x380 [10681.204707] ? end_repeat_nmi+0x16/0x1f [10681.204709] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10681.204710] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10681.204711] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10681.204713] [10681.204714] do_raw_spin_lock+0x177/0x190 [10681.204715] _raw_spin_lock+0x21/0x30 [10681.204716] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10681.204717] obd_mod_free+0x46/0x140 [ptlrpc] [10681.204718] mdc_commit_open+0x91/0xb0 [mdc] [10681.204719] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10681.204720] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10681.204721] after_reply+0xe2c/0x1780 [ptlrpc] [10681.204723] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10681.204724] ? __next_timer_interrupt+0x160/0x160 [10681.204725] ptlrpcd+0xa82/0xe40 [ptlrpc] [10681.204727] ? do_wait_intr+0xf0/0xf0 [10681.204728] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10681.204729] kthread+0x1d1/0x200 [10681.204730] ? set_kthread_struct+0x70/0x70 [10681.204731] ret_from_fork+0x1f/0x30 [10681.205832] rcu: rcu_sched kthread starved for 8169408 jiffies! g262441 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [10681.378019] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10681.384782] rcu: RCU grace-period kthread stack dump: [10681.386250] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10681.396814] Call Trace: [10681.400815] __schedule+0x351/0xcb0 [10681.402897] schedule+0xc0/0x180 [10681.405310] schedule_timeout+0xb4/0x190 [10681.407070] ? do_raw_spin_unlock+0x75/0x190 [10681.409187] ? __next_timer_interrupt+0x160/0x160 [10681.411692] ? prepare_to_swait_event+0xf3/0x140 [10681.413082] rcu_gp_kthread+0x8dd/0x15b0 [10681.416368] ? force_qs_rnp+0x2c0/0x2c0 [10681.417781] kthread+0x1d1/0x200 [10681.419038] ? set_kthread_struct+0x70/0x70 [10681.421303] ret_from_fork+0x1f/0x30 [10681.422306] rcu: Stack dump where RCU GP kthread last ran: [10681.424796] NMI backtrace for cpu 1 [10681.427529] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10681.434088] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10681.438445] Call Trace: [10681.439145] [10681.439728] ? dump_stack+0xbb/0x10e [10681.443852] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10681.447078] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10681.450054] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10681.453745] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10681.455240] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10681.458390] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [10681.462119] ? rcu_sched_clock_irq.cold.51+0x23f/0x307 [10681.467197] ? update_process_times+0x6d/0xc0 [10681.468414] ? tick_sched_handle.isra.18+0x27/0x90 [10681.472083] ? tick_sched_timer+0x53/0xb0 [10681.475429] ? __hrtimer_run_queues+0x1c2/0x4d0 [10681.479424] ? tick_sched_handle.isra.18+0x90/0x90 [10681.481606] ? hrtimer_interrupt+0x128/0x2c0 [10681.484965] ? smp_apic_timer_interrupt+0xa6/0x270 [10681.486967] ? apic_timer_interrupt+0xf/0x20 [10681.490047] [10681.490981] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10681.495527] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10681.497087] ? do_raw_spin_lock+0x177/0x190 [10681.498130] ? _raw_spin_lock+0x21/0x30 [10681.504859] ? ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10681.512950] ? ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10681.516658] ? ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10681.519771] ? ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10681.522442] ? ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10681.525412] ? woken_wake_function+0x30/0x30 [10681.527929] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10681.532325] ? kthread+0x1d1/0x200 [10681.533419] ? set_kthread_struct+0x70/0x70 [10681.536035] ? ret_from_fork+0x1f/0x30 [10688.060039] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [10688.065775] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [10688.066905] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10688.067144] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10688.067151] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10688.067157] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10688.073042] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10688.073132] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10688.073139] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10688.073142] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10688.073155] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10688.073159] RSP: 0018:ffffbc1d00d4bba8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10688.073172] RAX: 0000000000002231 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10688.073175] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10688.073178] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10688.073180] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10688.073182] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10688.073186] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10688.073189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10688.073192] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10688.073199] Call Trace: [10688.073205] [10688.073210] ? show_regs.cold.9+0x22/0x2f [10688.073219] ? watchdog_timer_fn.cold.6+0x62/0x10a [10688.073225] ? __hrtimer_run_queues+0x1c2/0x4d0 [10688.073229] ? watchdog+0x40/0x40 [10688.073233] ? hrtimer_interrupt+0x128/0x2c0 [10688.073237] ? smp_apic_timer_interrupt+0xa6/0x270 [10688.073246] ? apic_timer_interrupt+0xf/0x20 [10688.073250] [10688.073253] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10688.073258] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10688.073262] do_raw_spin_lock+0x177/0x190 [10688.073280] _raw_spin_lock+0x21/0x30 [10688.073285] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10688.073774] ? do_raw_spin_unlock+0x75/0x190 [10688.073929] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10688.074251] ? do_wait_intr+0xf0/0xf0 [10688.074259] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10688.075128] mdc_reint+0x2d/0x110 [mdc] [10688.075177] mdc_unlink+0x658/0x870 [mdc] [10688.075203] lmv_unlink+0x34d/0xbd0 [lmv] [10688.075231] ll_unlink+0x24d/0x7f0 [lustre] [10688.075369] vfs_unlink+0x202/0x330 [10688.075379] do_unlinkat+0x2fa/0x460 [10688.075384] __x64_sys_unlink+0x37/0x50 [10688.075387] do_syscall_64+0xc1/0x440 [10688.075394] entry_SYSCALL_64_after_hwframe+0x49/0xae [10688.075401] RIP: 0033:0x7f83e5f5c10b [10688.075409] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10688.075412] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10688.075765] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10688.075768] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10688.075770] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10688.075773] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10688.075775] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10688.348621] [10688.352437] RIP: 0010:native_safe_halt+0x2e/0x50 [10688.355251] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10688.367057] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10688.371904] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10688.375853] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10688.379452] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10688.383118] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10688.390234] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10688.393457] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10688.397617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10688.401571] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10688.404697] Call Trace: [10688.405688] [10688.407055] ? show_regs.cold.9+0x22/0x2f [10688.408591] ? watchdog_timer_fn.cold.6+0x62/0x10a [10688.410433] ? __hrtimer_run_queues+0x1c2/0x4d0 [10688.413620] ? watchdog+0x40/0x40 [10688.415229] ? hrtimer_interrupt+0x128/0x2c0 [10688.419067] ? smp_apic_timer_interrupt+0xa6/0x270 [10688.421071] ? apic_timer_interrupt+0xf/0x20 [10688.423080] [10688.423899] ? native_safe_halt+0x2e/0x50 [10688.425304] kvm_wait+0xcd/0xe0 [10688.426423] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10688.428226] do_raw_spin_lock+0x177/0x190 [10688.429707] _raw_spin_lock+0x21/0x30 [10688.430978] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10688.433036] process_one_work+0x2c8/0x700 [10688.434084] worker_thread+0x296/0x6e0 [10688.435411] ? rescuer_thread+0x570/0x570 [10688.436803] kthread+0x1d1/0x200 [10688.438034] ? set_kthread_struct+0x70/0x70 [10688.439227] ret_from_fork+0x1f/0x30 [10701.793060] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8188s! [10701.796456] Showing busy workqueues and worker pools: [10701.799162] workqueue events: flags=0x0 [10701.803573] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10701.811207] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10701.822482] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10701.833060] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10701.839438] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10701.844177] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10701.847703] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10701.851907] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10701.863249] workqueue events_long: flags=0x0 [10701.869753] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10701.874473] pending: lu_objects_flush [obdclass] [10701.880918] workqueue events_unbound: flags=0x2 [10701.895173] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10701.898042] pending: flush_memcg_stats_dwork, idle_cull_fn [10701.906682] workqueue events_power_efficient: flags=0x80 [10701.914115] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10701.930647] pending: neigh_periodic_work, do_cache_clean [10701.932699] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10701.939305] pending: neigh_periodic_work, check_lifetime [10701.948911] workqueue mm_percpu_wq: flags=0x8 [10701.954389] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10701.956864] pending: vmstat_update, lru_add_drain_per_cpu [10701.967181] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10701.973723] pending: vmstat_update [10701.979254] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10701.989405] pending: lru_add_drain_per_cpu BAR(48) [10701.992644] workqueue ipv6_addrconf: flags=0x40008 [10702.006432] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10702.013144] pending: addrconf_verify_work [10702.015889] workqueue knbd0-recv: flags=0x1a [10702.022280] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10702.032906] in-flight: 650:recv_work [10702.034752] workqueue ptlrpc_pinger: flags=0x2 [10702.037545] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10702.044975] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10702.049878] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8188s workers=4 idle: 36 466 29757 [10702.059208] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10692s workers=2 idle: 652 [10708.063607] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [10708.073956] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10708.152254] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10708.176416] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10708.190345] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10708.198903] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10708.233864] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10708.248636] RAX: 0000000000006e1a RBX: ffff9a61907912e0 RCX: 0000000000000001 [10708.261236] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10708.267106] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10708.275790] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10708.287089] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10708.301718] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10708.313945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10708.325729] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10708.334897] Call Trace: [10708.337994] [10708.338913] ? show_regs.cold.9+0x22/0x2f [10708.342189] ? watchdog_timer_fn.cold.6+0x62/0x10a [10708.348127] ? __hrtimer_run_queues+0x1c2/0x4d0 [10708.353858] ? watchdog+0x40/0x40 [10708.360491] ? hrtimer_interrupt+0x128/0x2c0 [10708.363211] ? smp_apic_timer_interrupt+0xa6/0x270 [10708.365864] ? apic_timer_interrupt+0xf/0x20 [10708.369721] [10708.370881] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10708.373857] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10708.375960] do_raw_spin_lock+0x177/0x190 [10708.377279] _raw_spin_lock+0x21/0x30 [10708.386754] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10708.395719] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10708.404872] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10708.410144] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10708.415817] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10708.427227] ? woken_wake_function+0x30/0x30 [10708.428592] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10708.439086] kthread+0x1d1/0x200 [10708.446572] ? set_kthread_struct+0x70/0x70 [10708.451619] ret_from_fork+0x1f/0x30 [10716.060030] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [10716.062083] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [10716.063347] Modules linked in: lustre(O) [10716.067204] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10716.067277] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10716.067285] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10716.067287] RIP: 0010:native_safe_halt+0x2e/0x50 [10716.067300] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10716.067305] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10716.067310] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10716.067312] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10716.067314] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10716.067316] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10716.067319] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10716.067321] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10716.067324] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10716.067326] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10716.067334] Call Trace: [10716.067339] [10716.067345] ? show_regs.cold.9+0x22/0x2f [10716.067354] ? watchdog_timer_fn.cold.6+0x62/0x10a [10716.067361] ? __hrtimer_run_queues+0x1c2/0x4d0 [10716.067365] ? watchdog+0x40/0x40 [10716.067369] ? hrtimer_interrupt+0x128/0x2c0 [10716.067374] ? smp_apic_timer_interrupt+0xa6/0x270 [10716.067380] ? apic_timer_interrupt+0xf/0x20 [10716.067385] [10716.067388] ? native_safe_halt+0x2e/0x50 [10716.067392] kvm_wait+0xcd/0xe0 [10716.067400] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10716.067407] do_raw_spin_lock+0x177/0x190 [10716.067411] _raw_spin_lock+0x21/0x30 [10716.067416] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10716.067685] ? do_raw_spin_unlock+0x75/0x190 [10716.067691] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10716.068143] ? do_wait_intr+0xf0/0xf0 [10716.068153] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10716.068358] mdc_reint+0x2d/0x110 [mdc] [10716.068397] mdc_unlink+0x658/0x870 [mdc] [10716.068424] lmv_unlink+0x34d/0xbd0 [lmv] [10716.068457] ll_unlink+0x24d/0x7f0 [lustre] [10716.068572] vfs_unlink+0x202/0x330 [10716.068582] do_unlinkat+0x2fa/0x460 [10716.068587] __x64_sys_unlink+0x37/0x50 [10716.068591] do_syscall_64+0xc1/0x440 [10716.068598] entry_SYSCALL_64_after_hwframe+0x49/0xae [10716.068604] RIP: 0033:0x7f83e5f5c10b [10716.068610] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10716.068614] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10716.068619] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10716.068621] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10716.068623] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10716.068625] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10716.068627] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10716.308058] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10716.326646] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10716.330605] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10716.333935] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10716.336661] RIP: 0010:native_safe_halt+0x2e/0x50 [10716.338519] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10716.344631] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10716.347393] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10716.349973] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10716.352782] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10716.355419] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10716.357731] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10716.360315] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10716.363467] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10716.365826] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10716.368464] Call Trace: [10716.369485] [10716.370121] ? show_regs.cold.9+0x22/0x2f [10716.371399] ? watchdog_timer_fn.cold.6+0x62/0x10a [10716.373268] ? __hrtimer_run_queues+0x1c2/0x4d0 [10716.374769] ? watchdog+0x40/0x40 [10716.375929] ? hrtimer_interrupt+0x128/0x2c0 [10716.377255] ? smp_apic_timer_interrupt+0xa6/0x270 [10716.378796] ? apic_timer_interrupt+0xf/0x20 [10716.380401] [10716.381171] ? native_safe_halt+0x2e/0x50 [10716.382682] kvm_wait+0xcd/0xe0 [10716.383686] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10716.385185] do_raw_spin_lock+0x177/0x190 [10716.386473] _raw_spin_lock+0x21/0x30 [10716.387827] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10716.389661] process_one_work+0x2c8/0x700 [10716.391193] worker_thread+0x296/0x6e0 [10716.392692] ? rescuer_thread+0x570/0x570 [10716.394335] kthread+0x1d1/0x200 [10716.395574] ? set_kthread_struct+0x70/0x70 [10716.397065] ret_from_fork+0x1f/0x30 [10728.056060] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ptlrpcd_rcv:2348] [10728.060640] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10728.108206] CPU: 0 PID: 2348 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10728.117450] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10728.131926] RIP: 0010:native_safe_halt+0x2e/0x50 [10728.135621] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10728.170368] RSP: 0018:ffffbc1d019bfbb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10728.184657] RAX: 0000000000000003 RBX: ffff9a61907912e0 RCX: ffff9a61c201e480 [10728.204890] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9a61907912e0 [10728.221652] RBP: ffff9a61c2034140 R08: 0000000000000008 R09: 0000000000000038 [10728.232419] R10: ffffffffab808b80 R11: ffffbc1d019bf8f0 R12: 0000000000040000 [10728.236993] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10728.243933] FS: 0000000000000000(0000) GS:ffff9a61c2000000(0000) knlGS:0000000000000000 [10728.250453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10728.258234] CR2: 0000560df275df44 CR3: 0000000078a16006 CR4: 0000000000170ef0 [10728.260728] Call Trace: [10728.264644] [10728.266440] ? show_regs.cold.9+0x22/0x2f [10728.269033] ? watchdog_timer_fn.cold.6+0x62/0x10a [10728.276330] ? __hrtimer_run_queues+0x1c2/0x4d0 [10728.280636] ? watchdog+0x40/0x40 [10728.282274] ? hrtimer_interrupt+0x128/0x2c0 [10728.291249] ? smp_apic_timer_interrupt+0xa6/0x270 [10728.296774] ? apic_timer_interrupt+0xf/0x20 [10728.301837] [10728.303042] ? native_safe_halt+0x2e/0x50 [10728.309073] kvm_wait+0xcd/0xe0 [10728.312923] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10728.318184] do_raw_spin_lock+0x177/0x190 [10728.321911] _raw_spin_lock+0x21/0x30 [10728.324781] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10728.331236] obd_mod_free+0x46/0x140 [ptlrpc] [10728.336899] mdc_commit_open+0x91/0xb0 [mdc] [10728.340677] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10728.346055] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10728.352571] after_reply+0xe2c/0x1780 [ptlrpc] [10728.358961] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10728.366864] ? __next_timer_interrupt+0x160/0x160 [10728.370803] ptlrpcd+0xa82/0xe40 [ptlrpc] [10728.375034] ? do_wait_intr+0xf0/0xf0 [10728.380043] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10728.385962] kthread+0x1d1/0x200 [10728.390803] ? set_kthread_struct+0x70/0x70 [10728.395136] ret_from_fork+0x1f/0x30 [10732.513598] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8219s! [10732.526338] Showing busy workqueues and worker pools: [10732.531500] workqueue events: flags=0x0 [10732.545895] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10732.569452] pending: osc_grant_work_handler [osc], push_to_pool, sec_gc_main [ptlrpc] [10732.600232] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10732.614628] pending: nfs4_renew_state [nfsv4], kfree_rcu_monitor [10732.624093] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10732.642019] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [10732.667312] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10732.676745] pending: vmstat_shepherd, cpu_latency_work [ptlrpc], key_garbage_collector [10732.687758] workqueue events_long: flags=0x0 [10732.690916] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10732.702928] pending: lu_objects_flush [obdclass] [10732.706870] workqueue events_unbound: flags=0x2 [10732.713941] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10732.725878] pending: flush_memcg_stats_dwork, idle_cull_fn [10732.734492] workqueue events_power_efficient: flags=0x80 [10732.743606] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10732.753584] pending: neigh_periodic_work, do_cache_clean [10732.761944] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10732.774314] pending: neigh_periodic_work, check_lifetime [10732.782022] workqueue mm_percpu_wq: flags=0x8 [10732.788956] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10732.799270] pending: vmstat_update, lru_add_drain_per_cpu [10732.805488] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10732.813777] pending: vmstat_update [10732.819137] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=3 [10732.831883] pending: lru_add_drain_per_cpu BAR(48) [10732.844353] workqueue ipv6_addrconf: flags=0x40008 [10732.854308] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10732.870827] pending: addrconf_verify_work [10732.879285] workqueue knbd0-recv: flags=0x1a [10732.881887] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10732.889164] in-flight: 650:recv_work [10732.897628] workqueue ptlrpc_pinger: flags=0x2 [10732.899183] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10732.911413] in-flight: 8:ptlrpc_pinger_main [ptlrpc] [10732.926949] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8219s workers=4 idle: 36 466 29757 [10732.937094] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10722s workers=2 idle: 652 [10736.065728] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ldlm_bl_01:15916] [10736.075651] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10736.145406] CPU: 1 PID: 15916 Comm: ldlm_bl_01 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10736.153780] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10736.166828] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10736.170439] Code: 4c 03 2c c5 80 58 68 ab 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10736.202610] RSP: 0018:ffffbc1d01cb7d58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10736.213633] RAX: 0000000000003383 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10736.219955] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9a61c20b4154 [10736.233459] RBP: ffff9a61c20b4140 R08: 0000000000000000 R09: 0000000000000000 [10736.245580] R10: 0000000003a4d419 R11: ffff9a61a052e000 R12: 0000000000080000 [10736.254732] R13: ffff9a61c2134140 R14: 000000000001e480 R15: ffff9a61c209e480 [10736.258525] FS: 0000000000000000(0000) GS:ffff9a61c2080000(0000) knlGS:0000000000000000 [10736.268915] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10736.278904] CR2: 00007fd5d7690000 CR3: 0000000078a16001 CR4: 0000000000170ee0 [10736.291384] Call Trace: [10736.292299] [10736.297215] ? show_regs.cold.9+0x22/0x2f [10736.303052] ? watchdog_timer_fn.cold.6+0x62/0x10a [10736.304759] ? __hrtimer_run_queues+0x1c2/0x4d0 [10736.311604] ? watchdog+0x40/0x40 [10736.317127] ? hrtimer_interrupt+0x128/0x2c0 [10736.322671] ? smp_apic_timer_interrupt+0xa6/0x270 [10736.328511] ? apic_timer_interrupt+0xf/0x20 [10736.340363] [10736.341672] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10736.356236] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10736.366922] do_raw_spin_lock+0x177/0x190 [10736.371760] _raw_spin_lock+0x21/0x30 [10736.388690] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10736.398310] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10736.410211] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10736.424517] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10736.431673] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10736.439345] ? woken_wake_function+0x30/0x30 [10736.451960] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10736.461536] kthread+0x1d1/0x200 [10736.463657] ? set_kthread_struct+0x70/0x70 [10736.474011] ret_from_fork+0x1f/0x30 [10744.060024] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:8] [10744.062027] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [writemany:36352] [10744.063030] Modules linked in: [10744.066952] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss [10744.068344] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10744.068414] CPU: 2 PID: 8 Comm: kworker/u8:0 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10744.080681] nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10744.080724] CPU: 3 PID: 36352 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10744.080740] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10744.080743] RIP: 0010:native_safe_halt+0x2e/0x50 [10744.080757] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10744.080761] RSP: 0018:ffffbc1d00d4bb98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10744.080765] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10744.080768] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c21b4154 [10744.080770] RBP: ffff9a61c21b4140 R08: 0000000000000000 R09: 0000000000000000 [10744.080771] R10: 00000000039e2a95 R11: ffff9a6182dfb000 R12: 0000000000100000 [10744.080774] R13: ffff9a61c2034140 R14: 000000000001e480 R15: ffff9a61c219e480 [10744.080777] FS: 00007f83e6722740(0000) GS:ffff9a61c2180000(0000) knlGS:0000000000000000 [10744.080780] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10744.080782] CR2: 00007fadbecb5e20 CR3: 000000012050e003 CR4: 0000000000170ee0 [10744.080793] Call Trace: [10744.080799] [10744.080804] ? show_regs.cold.9+0x22/0x2f [10744.080820] ? watchdog_timer_fn.cold.6+0x62/0x10a [10744.080827] ? __hrtimer_run_queues+0x1c2/0x4d0 [10744.080832] ? watchdog+0x40/0x40 [10744.080836] ? hrtimer_interrupt+0x128/0x2c0 [10744.080839] ? smp_apic_timer_interrupt+0xa6/0x270 [10744.080846] ? apic_timer_interrupt+0xf/0x20 [10744.080851] [10744.080854] ? native_safe_halt+0x2e/0x50 [10744.080863] kvm_wait+0xcd/0xe0 [10744.080871] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10744.080878] do_raw_spin_lock+0x177/0x190 [10744.080882] _raw_spin_lock+0x21/0x30 [10744.080888] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10744.081178] ? do_raw_spin_unlock+0x75/0x190 [10744.081184] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10744.081405] ? do_wait_intr+0xf0/0xf0 [10744.081415] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10744.081818] mdc_reint+0x2d/0x110 [mdc] [10744.081904] mdc_unlink+0x658/0x870 [mdc] [10744.081944] lmv_unlink+0x34d/0xbd0 [lmv] [10744.081981] ll_unlink+0x24d/0x7f0 [lustre] [10744.082109] vfs_unlink+0x202/0x330 [10744.082120] do_unlinkat+0x2fa/0x460 [10744.082126] __x64_sys_unlink+0x37/0x50 [10744.082129] do_syscall_64+0xc1/0x440 [10744.082142] entry_SYSCALL_64_after_hwframe+0x49/0xae [10744.082149] RIP: 0033:0x7f83e5f5c10b [10744.082159] Code: f0 ff ff 73 01 c3 48 8b 0d 7a 7d 39 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 7d 39 00 f7 d8 64 89 01 48 [10744.082163] RSP: 002b:00007fff7d2bcff8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [10744.082174] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f83e5f5c10b [10744.082176] RDX: 0000000000000000 RSI: 00007fff7d2bcfe4 RDI: 00007fff7d2bd060 [10744.082178] RBP: 00007fff7d2bd460 R08: 00007f83e62f423c R09: 00007f83e62f4260 [10744.082180] R10: 00007fff7d2bcda2 R11: 0000000000000246 R12: 0000000000000003 [10744.082182] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [10744.259039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10744.263374] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10744.266641] RIP: 0010:native_safe_halt+0x2e/0x50 [10744.269180] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10744.278452] RSP: 0018:ffffbc1d00c97dc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10744.282107] RAX: 0000000000000001 RBX: ffff9a61907912e0 RCX: 0000000000000001 [10744.285602] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9a61c2134154 [10744.288483] RBP: ffff9a61c2134140 R08: 0000000000000000 R09: 0000000000000000 [10744.291163] R10: 00000000039e3971 R11: ffff9a61a09b6000 R12: 00000000000c0000 [10744.295637] R13: ffff9a61c21b4140 R14: 000000000001e480 R15: ffff9a61c211e480 [10744.300945] FS: 0000000000000000(0000) GS:ffff9a61c2100000(0000) knlGS:0000000000000000 [10744.306109] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10744.308876] CR2: 000055a210266de8 CR3: 0000000078a16003 CR4: 0000000000170ee0 [10744.315206] Call Trace: [10744.316137] [10744.317126] ? show_regs.cold.9+0x22/0x2f [10744.319611] ? watchdog_timer_fn.cold.6+0x62/0x10a [10744.321780] ? __hrtimer_run_queues+0x1c2/0x4d0 [10744.324090] ? watchdog+0x40/0x40 [10744.325878] ? hrtimer_interrupt+0x128/0x2c0 [10744.328401] ? smp_apic_timer_interrupt+0xa6/0x270 [10744.330498] ? apic_timer_interrupt+0xf/0x20 [10744.332245] [10744.333266] ? native_safe_halt+0x2e/0x50 [10744.336042] kvm_wait+0xcd/0xe0 [10744.337262] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10744.339592] do_raw_spin_lock+0x177/0x190 [10744.341317] _raw_spin_lock+0x21/0x30 [10744.343110] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10744.345698] process_one_work+0x2c8/0x700 [10744.347134] worker_thread+0x296/0x6e0 [10744.348886] ? rescuer_thread+0x570/0x570 [10744.350138] kthread+0x1d1/0x200 [10744.351848] ? set_kthread_struct+0x70/0x70 [10744.353722] ret_from_fork+0x1f/0x30